Unlike Unicorn, Puma doesn't have a builtin HTTP request timeout
mechanism, so we have to use Rack::Timeout instead.
See the caveats in the Rack::Timeout documentation [1]. In Unicorn, a
timeout would send a SIGKILL to the worker, immediately killing it. This
would result in a dropped connection and a Cloudflare 502 error to the
user. In Puma, it raises an exception, which we can catch and return a
better error to the user. On the other hand, raising an exception can
potentially corrupt application state if it's sent at the wrong time, or
be delayed indefinitely if the app is stuck in IO or C extension code.
The default request timeout is 65 seconds. 65 seconds is to give things
like HTTP requests on a 60 second timeout enough time to complete. Set
the RACK_REQUEST_TIMEOUT environment variable to change the timeout.
1: https://github.com/sharpstone/rack-timeout#further-documentation
Add a model for storing image and video metadata for uploaded files.
Metadata is extracted using ExifTool. You will need to install ExifTool
after this commit. ExifTool 12.22 is the minimum required version
because we use the `--binary` option, which was added in this release.
The MediaMetadata model is separate from the MediaAsset model because
some files contain tons of metadata, and most of it is non-essential.
The MediaAsset model represents an uploaded file and contains essential
metadata, like the file's size and type, while the MediaMetadata model
represents all the other non-essential metadata associated with a file.
Metadata is stored as a JSON column in the database.
ExifTool returns all the file's metadata, not just the EXIF metadata.
EXIF is one of several types of image metadata, hence why we call
it MediaMetadata instead of EXIFMetadata.
A MediaAsset represents an image or video file uploaded to Danbooru. It
stores the metadata associated with the image or video. This is to work
on decoupling files from posts so that images can be uploaded separately
from posts.
DEPRECATION WARNING: `.reorder(nil)` with `.first` / `.first!` no
longer takes non-deterministic result in Rails 6.2. To continue
taking non-deterministic result, use `.take` / `.take!` instead.
(called from random at /home/user/src/danbooru/app/controllers/posts_controller.rb:91)
* Add README files to several directories in app/ giving a brief
overview of some parts of Danbooru's architecture.
* Add documentation for files in config/.
Replace the old IQDB API client with a new client for the new forked
version of IQDB at https://github.com/danbooru/iqdb.
Changes:
* The /iqdb_queries endpoint now returns `hash` and `signature` fields.
The `signature` is the full decoded Haar signature, while the `hash`
is a encoded version of the signature.
* The /iqdb_queries endpoint no longer returns `width` and `height`
fields in the response (these were always 128x128).
* We no longer need the IQDBs frontend server, now we talk to the IQDB
instance directly.
* We no longer send add/remove image commands to IQDB through AWS SQS,
now we send them to IQDB directly. They are sent in a delayed job so
that if IQDB is down, uploading images is still possible, the add
image commands will just get queued up.
* Fix a bug where regenerating an image's thumbnails didn't regenerate
IQDB, because IQDB silently ignored add image commands when the image
already existed in the database.
When a user tries to change their email, redirect them to the confirm
password page (like Github's sudo mode) instead of having them re-enter
their password on the change email page. This is the same thing we do
when a user updates their API keys. This way we have can use the same
confirm password authentication flow for everything that needs a
password.
Show the HTTP request headers and the client IP on the /status page.
This is for debugging request headers added by reverse proxies such as
Cloudflare and Nginx, and for making sure the client IP is correctly set
by the X-Forwarded-For header.
Allow admins to remove comment votes by other users. This is done by
clicking the comment score to get to the comment vote list, then
clicking the Remove button on every vote.
Make it so that when a user removes their own vote, the vote is soft
deleted (the is_deleted flag is set) instead of hard deleted.
Changes:
* Add is_deleted flag to comment votes.
* Relax uniqueness constraint so you can have multiple deleted votes on
the same comment. You can still only have one active vote on the comment.
* Add `soft_delete` method to Deletable concern.
Generate image URLs relative to the site's canonical URL instead of
relative to the domain of the current request.
This means that all subdomains of Danbooru - safebooru.donmai.us,
shima.donmai.us, saitou.donmai.us, and kagamihara.donmai.us - will use
image URLs from https://danbooru.donmai.us, instead of from the current
domain.
The main reason we did this before was so that we could generate either
http:// or https:// image URLs, depending on whether the current request
was HTTP or HTTPS, back when we tried to support both at the same time.
Now we support only HTTPS in production, so there's no need for this. It
was also pretty hacky, since it required storing the URL of the current
request in a per-request global variable in `CurrentUser`.
This also improves caching slightly, since users of safebooru.donmai.us
will receive cached images from danbooru.donmai.us.
Downstream boorus should make sure that the `canonical_url` and
`storage_manager` config options are set correctly. If you don't support
https:// in development, you should make sure to set the canonical_url
option to http:// instead of https://.
Changes:
* Change the `expires_at` field to `duration`.
* Make moderators choose from a fixed set of standard ban lengths,
instead of allowing arbitrary ban lengths.
* List `duration` in seconds in the /bans.json API.
* Dump bans to BigQuery.
Note that some old bans have a negative duration. This is because their
expiration date was before their creation date, which is because in 2013
bans were migrated to Danbooru 2 and the original ban creation dates
were lost.
Prevent saved searches, news updates, and ip bans from being publicly
dumped to BigQuery. They didn't override the `visible` method to
restrict their visibility for anonymous users.
Put the option to sticky a comment in the "..." popup menu instead of
in the comment edit form. This makes it more consistent with deleting or
undeleting a comment.
Also fix a bug where the comment undelete icon didn't show up due to a
typo.
Allow users to view their own rate limits with /rate_limits.json.
Note that rate limits are only updated after every API call, so this
page only shows the state of the limits after the last call, not the
current state.
* Tie rate limits to both the user's ID and their IP address.
* Make each endpoint have separate rate limits. This means that, for
example, your post edit rate limit is separate from your post vote
rate limit. Before all write actions had a shared rate limit.
* Make all write endpoints have rate limits. Before some endpoints, such
as voting, favoriting, commenting, or forum posting, weren't subject
to rate limits.
* Add stricter rate limits for some endpoints:
** 1 per 5 minutes for creating new accounts.
** 1 per minute for login attempts, changing your email address, or
for creating mod reports.
** 1 per minute for sending dmails, creating comments, creating forum
posts, or creating forum topics.
** 1 per second for voting, favoriting, or disapproving posts.
** These rate limits all have burst factors high enough that they
shouldn't affect normal, non-automated users.
* Raise the default write rate limit for Gold users from 2 per second to
4 per second, for all other actions not listed above.
* Raise the default burst factor to 200 for all other actions not listed
above. Before it was 10 for Members, 30 for Gold, and 60 for Platinum.
Rework the rate limit implementation to make it more flexible:
* Allow setting different rate limits for different actions. Before we
had a single rate limit for all write actions. Now different
controller endpoints can have different limits.
* Allow actions to be rate limited by user ID, by IP address, or both.
Before actions were only limited by user ID, which meant non-logged-in
actions like creating new accounts or attempting to login couldn't be rate
limited. Also, because actions were limited by user ID only, you could
use multiple accounts with the same IP to get around limits.
Other changes:
* Remove the API Limit field from user profile pages.
* Remove the `remaining_api_limit` field from the `/profile.json` endpoint.
* Rename the `X-Api-Limit` header to `X-Rate-Limit` and change it from a
number to a JSON object containing all the rate limit info
(including the refill rate, the burst factor, the cost of the call,
and the current limits).
* Fix a potential race condition where, if you flooded requests fast
enough, you could exceed the rate limit. This was because we checked
and updated the rate limit in two separate steps, which was racy;
simultaneous requests could pass the check before the update happened.
The new code uses some tricky SQL to check and update multiple limits
in a single statement.
Add a new color palette and rework all site colors (both light mode and dark mode) to
use the new palette.
This ensures that colors are used consistently, from a carefully designed color palette,
instead of being chosen at random.
Before, colors in light mode were chosen on an ad-hoc basis, which resulted in a lot of
random colors and inconsistent design.
The new palette has 7 hues: red, orange, yellow, green, blue, azure (a lighter blue), and
purple. There's also a greyscale. Each hue has 10 shades of brightness, which (including
grey) gives us 80 total colors.
Colors are named like this:
var(--red-0); /* very light red */
var(--red-2); /* light red */
var(--red-5); /* medium red */
var(--red-7); /* dark red */
var(--red-9); /* very dark red */
var(--green-7); /* dark green */
var(--blue-5); /* medium blue */
var(--purple-3); /* light purple */
/* etc */
The color palette is designed to meet the following criteria:
* To have close equivalents to the main colors used in the old color scheme,
especially tag colors, so that changes to major colors are minimized.
* To produce a set of colors that can be used as as main text colors, as background
colors, and as accent colors, both in light mode and dark mode.
* To ensure that colors at the same brightness level have the same perceived brightness.
Green-4, blue-4, red-4, purple-4, etc should all have the same brightness and contrast
ratios. This way colors look balanced. This is actually a difficult problem, because human
color perception is non-linear, so you can't just scale brightness values linearly.
There's a color palette test page at https://danbooru.donmai/static/colors
Notable changes to colors in light mode:
* Username colors are the same as tag colors.
* Copyright tags are a deeper purple.
* Builders are a deeper purple (fixes#4626).
* Moderators are green.
* Gold users are orange.
* Parent borders are a darker green.
* Child borders are a darker orange.
* Unsaved notes have a thicker red border.
* Selected notes have a thicker blue (not green) border.
Remove the api_token field from the response to the login action (POST
/sessions). This doesn't make sense in the presence of multiple API
keys, and is also not generally useful; if you need an API key, create
one yourself and write it down.
Require the user to re-enter their password before they can view,
create, update, or delete their API keys.
This works by tracking the timestamp of the user's last password
re-entry in a `last_authenticated_at` session cookie, and redirecting
the user to a password confirmation page if they haven't re-entered
their password in the last hour.
This is modeled after Github's Sudo mode.
Add the ability to restrict API keys so that they can only be used with
certain IP addresses or certain API endpoints.
Restricting your key is useful to limit damage in case it gets leaked or
stolen. For example, if your key is on a remote server and it gets
hacked, or if you accidentally check-in your key to Github.
Restricting your key's API permissions is useful if a third-party app or
script wants your key, but you don't want to give full access to your
account.
If you're an app or userscript developer, and your app needs an API key
from the user, you should only request a key with the minimum
permissions needed by your app.
If you have a privileged account, and you have scripts running under
your account, you are highly encouraged to restrict your key to limit
damage in case your key gets leaked or stolen.
* Add an explanation of what an API key is and how to use it.
* Make it possible for the site owner to view all API keys.
* Remove the requirement to re-enter your password before you can view
your API key (to be reworked).
* Move the API key controller from maintenance/user/api_keys_controller.rb
to a top level controller.
6d867de20 caused an exception in the ApiKeysController, which calls
respond_with with two arguments: `respond_with(CurrentUser.user, @api_key)`.
`options[0]` referred to the second argument, which was incorrect.
Like 9efb374ae, allow users to toggle between upvoting and downvoting a
post without raising an error or having to manually remove the vote
first. If you upvote a post, then downvote it, the upvote is
automatically removed and replaced by the downvote.
Other changes:
* Tagging a post with `upvote:self` or `downvote:self` is now silently
ignored when the user doesn't have permission to vote, instead of
raising an error.
* Undoing a vote that doesn't exist now does nothing instead of
returning an error. This can happen if you open the same post in two
tabs, undo the vote in tab 1, then try to undo the vote again in tab 2.
Changes to the /post_votes API:
* `POST /post_votes` and `DELETE /post_votes` now return a post vote
instead of a post.
* The `score` param in `POST /post_votes` is now 1 or -1, not `up` or
`down`.
The /emails endpoint was passing in the "Email" model because that's
how the emails controller classifies. This was to fix that, and to
allow any other such cases in the future.
Allow users to upvote a comment, then downvote it, without raising an
error or having to manually remove the upvote first. The upvote is
automatically removed and replaced by the downvote.
Changes to the /comment_votes API:
* `POST /comment_votes` and `DELETE /comment_votes` now return a comment
vote instead of a comment.
* The `score` param in `POST /comment_votes` is now 1 or -1, not
`up` or `down.`
Previously thresholded comments were hidden completely. You had to click
the "Show X hidden comments" button to unhide all hidden comments in a
thread. Now it works like this:
* When a comment is below your threshold, the comment text is hidden and
replaced by a `[hidden]` link, which you can click to unhide the comment.
* When a comment is at half your threshold (for example, your threshold
is -8 but the comment is at -4), then the comment is greyed out.
This means that comments aren't completely hidden, they're just
collapsed, so you can see the commenter and the score without unhiding
the comment. It also means you don't have to scroll back up to unhide a
comment, and threads aren't disrupted by comments being secretly
hidden (which is confusing when people are replying to hidden comments,
which forces you to go back up and unhide to find).
This refactors Pundit policies to only rely on the current user, not on
the current user and the current HTTP request. In retrospect, it was a
bad idea to include the current request in the Pundit context. It bleeds
out everywhere and there are many contexts (in tests and models) where
we only have the current user, not the current request. The previous
commit got rid of the only two places where we used it.
Specify the default settings for new users inside the User model instead
of inside the database. This makes it easier to change defaults, and it
makes the code clearer.
