Break the hourly/daily/weekly/monthly maintenance tasks down into
individual delayed jobs. This way if one task fails, it won't prevent
other tasks from running. Also, jobs can be run in parallel, and can be
individually retried if they fail.
Move the metadata parsing code from MediaAsset to ExifTool::Metadata so
we can use it outside the context of a MediaAsset, in particular when
dealing with a MediaFile that hasn't been saved to disk yet.
Set `crossorigin="anonymous"` on the <img> tag for thumbnails. This
makes it so we don't send session cookies in requests for thumbnails.
Since images are served from cdn.donmai.us, and since session cookies
are set on *.donmai.us, session cookies are sent by default with every
thumbnail request.
Not sending cookies saves up to 1kb of overhead per thumbnail request
(in reality, it's less than this because of HTTP/2 multiplexing and
header compression).
Caused by d854bf6b. Banning an artist would deadlock because it was
performed in a transaction, which didn't work with the `parallel_each`
inside the "create an implication to banned_artist" step.
Fix a bug where where PNG images could be incorrectly detected as
exif-rotated. This would happen when a PNG contained the
IFD0:Orientation flag. It's technically possible for a PNG to contain
this flag, but it's ignored by libvips and by browsers.
post #3762340 (nsfw) is an example of a PNG like this.
The fix is to use `autorot` to let libvips apply the rotation instead of
trying to interpret the exif data ourselves. Note that libvips-8.9 has a
bug where it doesn't strip the orientation flag after applying
`autorot`, which leads to the image being incorrectly rotated a second
time when generating the thumbnail. Use libvips-8.11 instead.
Hourly pruning of expired uploads was failing because of nil deference
errors in `media_asset.destroy!`. There are various cases where an
upload doesn't have a media asset, for example when the source url
fails to download or when the upload is of an invalid filetype.
`IFD0:Orientation` is the orientation of the main image.
`IFD1:Orientation` is the orientation of the embedded thumbnail, if it
has one. Using `IFD1:Orientation` was incorrect here because some images
have a non-rotated main image but a rotated thumbnail. Post #1023563 is
an example.
Rotate the image based on the EXIF orientation flag when generating
thumbnails and samples.
Also fix the width and height to be calculated correctly for rotated
images. Vips gives us the unrotated width and height of the image; we
have to detect whether the image is rotated and swap the width and
height manually to correct them. For example, if an image with the
"Rotate 90 CW" flag is 100x500 before rotation, then after rotation it's
500x100. This should fix#4883 (Exif rotation breaks Javascript fit-to-window)
We also have to fix it so that regenerating a post updates the width and
height of the post, in the event that it's a rotated image.
Finally we set `image-orientation: from-image;` even though it's
probably not necessary.
Make it so that when ARCHIVE_DATABASE_URL isn't set, it defaults to
DATABASE_URL. In other words, if you don't have a separate archive
database configured, then default to using the main database for
post/pool versions.
Fixes an issue where running the test suite would fail if you didn't
explicitly set ARCHIVE_DATABASE_URL because it tried to use
`archive_test` as the post/pool versions database name.
Autotag `greyscale`, `non-repeating_animation`, and `exif_rotation`.
Note that this does not detect all (or even most) greyscale images.
Artists often save greyscale images as RGB instead of as greyscale.
Automatically tag animated_gif and animated_png when a post is edited.
Add them back if the user tries to remove them from an animated post,
or remove them if the user tries to add them to a non-animated post.
Before we added these tags at upload time, but it was possible for users
to remove them after upload, or to incorrectly add them to non-animated
posts. They were added at upload time because we couldn't afford to open
the file and parse the metadata on every tag edit. Now that we save the
metadata in the database, we can do this.
This also makes it so you can't tag ugoira on non-ugoira files.
Known bug: it's possible to have an animated GIF where every frame is
identical. Post #3770975 is an example. This will be detected as an
animated GIF even though visually it doesn't appear to be animated.
Fixes#4041: Animated_gif tag not added to preprocessed uploads
Remove code for working with older versions of libvips. This makes
libvips 8.10+ a hard requirement. Older versions were already broken and
failed certain tests in the test suite.
Fix it so that you can reapprove a failed BUR to run it again. Before
this would fail because it would end up trying to create the aliases
or implications again, which would fail because they already existed.
Now it ignores when an alias or implication already exists. It will
however finish tagging the posts if they haven't been fully moved.
The artist ban tests deadlocked because of a weird interaction between
threads and database transactions when tagging posts in parallel. Add a
hack to work around it.
When processing an alias, rename, implication, mass update, or nuke,
update the posts in parallel. This means that if we alias foo to bar,
for example, then we use four processes at once to retag the posts from
foo to bar.
This doesn't mean that if we have two aliases in a BUR, we process both
aliases in parallel. It simply means that when processing an alias, we
update the posts in parallel for that alias.
When a BUR is approved, put it in a `processing` state. After it
successfully finishes processing, put it in the `approved` state. If it
fails processing, put it in the `failed` state.
If approving the BUR fails with a validation error, for example because
the alias already exists or an implication lacks a wiki, then leave the
BUR in the `pending` state. The `failed` state is only for unexpected
errors during processing.
Change the way BURs are processed. Before, we spawned a background job
for each line of the BUR, then processed each job sequentially. Now, we
process the entire BUR sequentially in a single background job.
This means that:
* BURs are truly sequential now. Before certain things like removing
aliases weren't actually performed in a background job, so they were
performed out-of-order before everything else in the BUR.
* Before, if an alias or implication line failed, then subsequent alias
or implication lines would still be processed. This was because each
alias or implication line was queued as a separate job, so a failure
of one job didn't block another. Now, if any alias or implication
fails, the entire BUR will fail and stop processing after that line.
This may be good or bad, depending on whether we actually need the BUR
to be processed in order or not.
* Before, BURs were processed inside a database transaction (except for the
actual updating of posts). Now they're not. This is because we can't
afford to hold transactions open while processing long-running aliases
or implications. This means that if BUR fails in the middle when it is
initially approved, it will be left in a half-complete state. Before
it would be rolled back and left in a pending state with no changes
performed.
* Before, only one BUR at a time could be processed. If multiple BURs
were approved at the same time, then they would queue up and be
processed one at a time. Now, multiple BURs can be processed at the
same time. This may be undesirable when processing large BURs, or BURs
that must be approved in a specific order.
* Before, large tag category changes could time out. This was because
they weren't actually performed in a background job. Now they are, so
they shouldn't time out.
Bug: if ExifTool exited with status 1 because it thought the file was
corrupt, then we didn't record any of the metadata, even though it was
able to read most of it. It turns out there are thousands of posts with
minorly corrupt metadata that ExifTool is still able to read, but will
complain about.
Fix: ignore the exit code of ExifTool and always save whatever metadata
ExifTool is able to return. It will return an `ExifTool:Error` tag in
the event of errors.
Note that there are some (many?) files that are considered corrupt by
ExifTool but not by Vips, and vice versa. Probably because ExifTool only
parses the metadata while Vips only parses the image data.
Unlike Unicorn, Puma doesn't have a builtin HTTP request timeout
mechanism, so we have to use Rack::Timeout instead.
See the caveats in the Rack::Timeout documentation [1]. In Unicorn, a
timeout would send a SIGKILL to the worker, immediately killing it. This
would result in a dropped connection and a Cloudflare 502 error to the
user. In Puma, it raises an exception, which we can catch and return a
better error to the user. On the other hand, raising an exception can
potentially corrupt application state if it's sent at the wrong time, or
be delayed indefinitely if the app is stuck in IO or C extension code.
The default request timeout is 65 seconds. 65 seconds is to give things
like HTTP requests on a 60 second timeout enough time to complete. Set
the RACK_REQUEST_TIMEOUT environment variable to change the timeout.
1: https://github.com/sharpstone/rack-timeout#further-documentation
Change this message:
2 post(s) on this page were hidden by safe mode. Go to Danbooru or
disable safe mode to view them (learn more).
To link to [[help:safe mode]] instead of [[help:user settings]].
Fix a bug where if you did a slow search that took too long to calculate
the page count, and you had 200 posts per page, then we would show page
5000 as the last page of the search.
This was because we were artificially returning 1,000,000 as the post
count to signal that the count timed out, but at 200 posts per page this
would show 5000 as the last page of the search.
