Changes:
* Change the `expires_at` field to `duration`.
* Make moderators choose from a fixed set of standard ban lengths,
instead of allowing arbitrary ban lengths.
* List `duration` in seconds in the /bans.json API.
* Dump bans to BigQuery.
Note that some old bans have a negative duration. This is because their
expiration date was before their creation date, which is because in 2013
bans were migrated to Danbooru 2 and the original ban creation dates
were lost.
Remove the `category_name` field from the `/wiki_page.json` API. This
field was originally added only because it was needed by our autocomplete
Javascript. It was also misnamed, it wasn't the tag's category name, it
was the category's ID.
Users should use `https://danbooru.donmai.us/wiki_pages.json?only=title,tag`
instead if they need this.
This triggered a N+1 query pattern when dumping wiki pages to BigQuery,
which made dumping wiki pages very slow. It also meant this field was
included in the database dump, even though it wasn't a real database
column.
Prevent saved searches, news updates, and ip bans from being publicly
dumped to BigQuery. They didn't override the `visible` method to
restrict their visibility for anonymous users.
* Export daily public database dumps to BigQuery and Google Cloud Storage.
* Only data visible to anonymous users is exported. Some tables have
null or missing fields because of this.
* The bans table is excluded because some bans have an expires_at
timestamp set beyond year 9999, which BigQuery doesn't support.
* The favorites table is excluded because it's too slow to dump (it
doesn't have an id index, which is needed by find_each).
* Version tables are excluded because dumping them every day is
inefficient, streaming insertions should be used instead.
Links:
* https://console.cloud.google.com/bigquery?project=danbooru1
* https://console.cloud.google.com/storage/browser/danbooru_public
* https://storage.googleapis.com/danbooru_public/data/posts.json
* Max comment length: 15,000 characters.
* Max forum post length: 200,000 characters.
* Max forum topic title length: 200 characters.
* Max dmail length: 50,000 characters.
* Max dmail title length: 200 characters.
* Don't set the inviter field for newly promoted users, or for Gold/Plat
upgrades.
* Clear the inviter field for paid Gold/Plat upgrades, and for users who
have a feedback or a modaction listing who invited them. This leaves
about 600 remaining users with an inviter field with no other record
of who invited them.
See #4750.
If we detect that the session cookie has expired (by the presence of the
`#login_illust` element on the page), then clear the cached session
cookie. The current source fetch will still fail, but the next fetch
will try to login again and hopefully succeed.
* When trying to create an artist entry for a non-artist tag, set the
error on the name attribute so that the artist name gets marked
as incorrect in the artist edit form.
* Fix a bad `Name '' cannot be blank` error message when the artist name
is blank.
* Fix showing wiki pages of non-artist tags in the artist edit form when
the artist name conflicts with a non-artist tag (e.g. if you try to
create an artist named '1girl', don't show the wiki for 1girl in the
artist edit form).
Put the option to sticky a comment in the "..." popup menu instead of
in the comment edit form. This makes it more consistent with deleting or
undeleting a comment.
Also fix a bug where the comment undelete icon didn't show up due to a
typo.
Change error from "Body has no content" to "Body can't be blank" when a
user tries to submit an empty comment. This makes it consistent with
error messages in other models when someone tries to submit blank content.
Move the BUR help text from the <textarea> placeholder attribute to a
<details> tag that embeds the [[help:bur_notice]] wiki page.
Also update some CSS for the <details> tag that was meant to only apply
to the user upgrades page and didn't look good here.
Only use hover to hide the checkboxes on devices that support hovering
(i.e. computers with a mouse). On some mobile devices, a tap is used to
emulate hovering, which meant the tag had to be tapped twice.
Allow users to view their own rate limits with /rate_limits.json.
Note that rate limits are only updated after every API call, so this
page only shows the state of the limits after the last call, not the
current state.
If the user makes a request while rate limited, penalize them 1 second
for that request, up to a maximum of 30 seconds. This means that if a
user doesn't stop making requests after being rate limited, then they
will stay rate limited forever until they stop.
This is to temp ban bots, especially spam bots, that flood requests
while ignoring HTTP errors or rate limits.
(Note that this is on a per-endpoint basis. Being rate limited on one
endpoint won't penalize you for making calls to other endpoints.)
* Tie rate limits to both the user's ID and their IP address.
* Make each endpoint have separate rate limits. This means that, for
example, your post edit rate limit is separate from your post vote
rate limit. Before all write actions had a shared rate limit.
* Make all write endpoints have rate limits. Before some endpoints, such
as voting, favoriting, commenting, or forum posting, weren't subject
to rate limits.
* Add stricter rate limits for some endpoints:
** 1 per 5 minutes for creating new accounts.
** 1 per minute for login attempts, changing your email address, or
for creating mod reports.
** 1 per minute for sending dmails, creating comments, creating forum
posts, or creating forum topics.
** 1 per second for voting, favoriting, or disapproving posts.
** These rate limits all have burst factors high enough that they
shouldn't affect normal, non-automated users.
* Raise the default write rate limit for Gold users from 2 per second to
4 per second, for all other actions not listed above.
* Raise the default burst factor to 200 for all other actions not listed
above. Before it was 10 for Members, 30 for Gold, and 60 for Platinum.
Rework the rate limit implementation to make it more flexible:
* Allow setting different rate limits for different actions. Before we
had a single rate limit for all write actions. Now different
controller endpoints can have different limits.
* Allow actions to be rate limited by user ID, by IP address, or both.
Before actions were only limited by user ID, which meant non-logged-in
actions like creating new accounts or attempting to login couldn't be rate
limited. Also, because actions were limited by user ID only, you could
use multiple accounts with the same IP to get around limits.
Other changes:
* Remove the API Limit field from user profile pages.
* Remove the `remaining_api_limit` field from the `/profile.json` endpoint.
* Rename the `X-Api-Limit` header to `X-Rate-Limit` and change it from a
number to a JSON object containing all the rate limit info
(including the refill rate, the burst factor, the cost of the call,
and the current limits).
* Fix a potential race condition where, if you flooded requests fast
enough, you could exceed the rate limit. This was because we checked
and updated the rate limit in two separate steps, which was racy;
simultaneous requests could pass the check before the update happened.
The new code uses some tricky SQL to check and update multiple limits
in a single statement.
The search form uses a <button type="submit"> tag instead of a
<input type="submit"> tag. It uses a <button> because it contains an
icon and <input> tags can't contain nested tags.
Light mode:
* Change child post border from orange back to dark yellow (still darker
than previous yellow).
* Make flagged borders brighter red.
* Make admins brighter red.
* Make parent, child, and pending post notice bars brighter.
* Change copyright tags from purple to magenta (very close to copyright
tag color from before).
* Darken forum topic new/approved/rejected labels.
Dark mode:
* Make platinum users brighter grey.
Add site icons linking to all the artist's sites in the fetch source
data box.
Some artist entries have a large number of URLs. Various heuristics are
applied to try to present the most useful URLs first. Dead URLs and
redundant URLs (Pixiv stacc and Twitter intent URLs) are filtered out.
Remaining URLs are sorted first by site (to put sites like Pixiv and
Twitter first), then by URL (to break ties when an artist has multiple
accounts on the same site).
Some sites have shitty hard-to-read icons. It can't be helped. The icons
are the official favicons of each site.
This happens because toggling a related tag updates the tag box, which
triggers autocomplete, which can't find a tag because the cursor isn't
in the tag box, so it ends up making an autocomplete call for the empty
string.
Make related tags use a single-column layout on mobile.
Related tags layout is still broken inside the tag edit dialog box, when
the dialog box is too small for multiple columns.
* Use -webkit-text-stroke instead of bold to indicate selected tags.
Bold causes the tag to change width, which is jarring and can cause
the tag to wrap around to a new line. -webkit-text-stroke doesn't
change the tag's width. Despite the name, this property is widely
supported.
* Don't wrap the checkbox in a label. In combination with flexbox, this
made the hitbox for the tag too big. You could click on empty space
outside the tag and it would still toggle the tag.
Instead of changing the background color of the entire page when you
change modes in the post mode menu (which was ugly, and not obvious what
the colors meant), show a notice and make it so that hovering a post
gives it a blue outline. This indicates that clicking the post will do
something different.
Fix issues with note sizing caused by the switch to `box-sizing: border-box`
in 19aea560. Note boxes have a 1px black border, which is counted as
part of the note box width when using border-box. This threw off sizing
and position calculations.
Also go back to 1px borders instead of 2px borders for unsaved and
selected notes. This also messed up note sizing.
* Standardize dialog box and tooltip box shadows.
* Add shadows behind all other floating elements, namely the
autocomplete menu and translation notes (except for embedded notes,
which are usually meant to be seamless).
Remove some rules intended to highlight the currently selected tag in
the autocomplete menu. The selected tag was supposed to be shown with
the tag hover color, but this didn't actually work because of CSS
specificity issues, and it possibly never worked to begin with, so it's
not worth fixing.
Partial revert of b9ea9d2f5. Go back to using a semi-transparent
background to indicate the selected parent/child post, but add more
padding and adjust the transparency to make it more legible.
In the related tags list, use checkboxes and bold to indicate selected
tags, instead of highlighting selected tags with a blue background.
This is so that you can see the colors of selected tags in the related
tags list, and in particular so you can see the artist tag.
Change the new, approved, pending, and rejected labels in the forum to
use an outlined pill style (like user levels in user tooltips) instead
of a filled background style. This is less obtrusive and more readable
in dark mode.
Remove the follow properties:
* clear (wasn't used)
* list-style-* (wasn't used)
* outline (only used a few times; can be achieved with borders or
box-shadow).
* mask (not well-supported in browsers; only used a few times for
effects that could be achieved in other ways).
* text-decoration-* (use the shorthand property)
* text-transform (can be achieved in other ways)
* Fix it so that all edit forms show an error banner if the form
has validation errors. Previously forms had to manually call
`error_messages_for`, which not all forms did.
* Fix it so that the full validation error message is shown next to each
input attribute that had errors. Also update the styling of these
error messages to look better.
* On the post show page, in the parent/child box, change it so that
currently selected post is more visible. Use a light blue outline
around the current post instead of a dark semi-transparent border.
* Fix it so that there's less empty space around thumbnails in the
parent/child box.
Add a new color palette and rework all site colors (both light mode and dark mode) to
use the new palette.
This ensures that colors are used consistently, from a carefully designed color palette,
instead of being chosen at random.
Before, colors in light mode were chosen on an ad-hoc basis, which resulted in a lot of
random colors and inconsistent design.
The new palette has 7 hues: red, orange, yellow, green, blue, azure (a lighter blue), and
purple. There's also a greyscale. Each hue has 10 shades of brightness, which (including
grey) gives us 80 total colors.
Colors are named like this:
var(--red-0); /* very light red */
var(--red-2); /* light red */
var(--red-5); /* medium red */
var(--red-7); /* dark red */
var(--red-9); /* very dark red */
var(--green-7); /* dark green */
var(--blue-5); /* medium blue */
var(--purple-3); /* light purple */
/* etc */
The color palette is designed to meet the following criteria:
* To have close equivalents to the main colors used in the old color scheme,
especially tag colors, so that changes to major colors are minimized.
* To produce a set of colors that can be used as as main text colors, as background
colors, and as accent colors, both in light mode and dark mode.
* To ensure that colors at the same brightness level have the same perceived brightness.
Green-4, blue-4, red-4, purple-4, etc should all have the same brightness and contrast
ratios. This way colors look balanced. This is actually a difficult problem, because human
color perception is non-linear, so you can't just scale brightness values linearly.
There's a color palette test page at https://danbooru.donmai/static/colors
Notable changes to colors in light mode:
* Username colors are the same as tag colors.
* Copyright tags are a deeper purple.
* Builders are a deeper purple (fixes#4626).
* Moderators are green.
* Gold users are orange.
* Parent borders are a darker green.
* Child borders are a darker orange.
* Unsaved notes have a thicker red border.
* Selected notes have a thicker blue (not green) border.
* Render counts in the user profile tooltip the same way as tag counts.
* Render tag counts in the tags index page the same way as tag counts in
the tags list.
Bug: In Postgres 13, getting the count of a blank search underestimated
the page count by a large margin (~700,000 posts).
The query we were executing was this:
EXPLAIN (FORMAT JSON) SELECT * FROM posts ORDER BY id DESC
The `ORDER BY id DESC` clause triggered a parallel seq scan query plan
in Postgres 13, which for some reason causes Postgres to underestimate
the row count by large amount in each parallel branch.
Getting rid of the ORDER BY clause makes it do a regular seq scan, which
gives an accurate estimate.
* Refactor the paginator into a ViewComponent.
* Fix inconsistent spacing between paginator items.
* Fix a bug where the sequential paginator generated the wrong next /
previous page links in the <link rel="{next|prev}"> tags in the <head>.
* Always include the final page as a hidden html element, so that it can
be unhidden with custom CSS.
* Make it easier to change the pagination window.
This makes it so that an element's width includes borders and padding,
so that borders or padding don't cause an element to exceed its
specified width. This is a standard part of most CSS resets.
https://developer.mozilla.org/en-US/docs/Web/CSS/box-sizing
Increase the font weight from 200 to 400 for the IBM Plex Mono font used
in notes. 400 is the standard weight we use for other fonts. 200 made
the font too thin and hard to read at normal font sizes.
Adjust input boxes to fit the screen better on mobile, and to use
consistent sizes throughout the site (no ad-hoc overrides to make
certain input boxes a bit bigger in random places).
On desktop, this makes most input boxes a bit taller and narrower. On
mobile, it makes input boxes the full width of the screen.
This most notably affects the tag edit box, the comment and forum
post box, the wiki page edit box, and the commentary box.
Refactor CSS to use standard Tailwind-style utility classes instead of
ad-hoc rules. This eliminates a lot of single-purpose rules for specific
UI elements and standardizes margins to be more consistent throughout
the site.
Utility classes are defined manually on an as-needed basis instead of
importing Tailwind as a whole. Naming conventions mostly follow
Tailwind's conventions, otherwise they follow Bootstrap.
* https://tailwindcss.com/docs/
* https://getbootstrap.com/docs/5.0/utilities/spacing/
evazion
evazion