Let all users have unlimited favorites. Formerly the limit was 10k
favorites for regular members, 20k for Gold, and unlimited for Platinum.
Limiting favorites doesn't make sense since upvotes are unlimited.
Stop using the pool_string attribute on posts:
* Stop updating it when adding or removing posts from pools.
* Stop returning pool_string in the /posts.json API.
* Stop including the `data-pools` attribute on thumbnails.
The pool_string attribute was used in the past to facilitate pool:X
searches. Posts had a hidden pool_string attribute that contained a list
of every pool the post belonged to. These pools were treated like fake
hidden tags on the post and a search for `pool:X` was treated like a tag
search.
The pool_string has no longer been used for this purpose for a long time
now, and was only maintained for API compatibility purposes. Getting rid
of it eliminates a bunch of legacy cruft relating to adding and removing
posts from pools.
If you need to see which pools a post belongs to, do this:
* https://danbooru.donmai.us/pools.json?search[post_ids_include_any]=318550
The `data-pools` attribute on thumbnails was used by some people to add
custom borders to pooled posts with custom CSS. This will no longer
work. This was already broken because it included things like collection
pools and deleted pools, which you probably didn't want. Use a
userscript to add this attribute back to thumbnails if you need it.
Start storing the duration of animations and videos in the `duration`
field on the media_assets table. This had to wait until 3d30bfd69 was
deployed, which had to wait until Postgres was upgraded in order to add
the duration column to the media_assets table without downtime.
Also add a fix script to backfill the duration on existing posts. Usage:
TAGS=animated ./script/fixes/079_fix_duration.rb
Fix certain animated PNGs returning NaN as the duration because the
frame rate was being reported as "0/0" by FFMpeg. This happens when the
animation has zero delay between frames. This is supposed to mean a PNG
with an infinitely fast frame rate, but in practice browsers limit it to
around 10FPS. The exact frame rate browsers will use is unknown and
implementation defined.
Fix regression in 2eb89a835 that broke the modqueue page because the
arguments to `paginated_search` changed and weren't updated here.
Also fix incorrect YARD documentation syntax.
* On /pools, hide deleted pools by default in HTML responses. Don't
filter out deleted pools in API responses.
* API change: on /forum_topics, only hide deleted forum topics by
default for HTML responses, not for API responses. Explicitly do
https://danbooru.donmai.us/forum_topics.json?search[is_deleted]=false
to filter out deleted topics.
* API change: on /tags, only hide empty tags by default for HTML
responses, not for API responses. Explicitly do
https://danbooru.donmai.us/tags.json?search[is_empty]=false to filter
out empty tags.
* API change: on /pools, default to 20 posts per page for API responses,
not 40.
* API change: add `search[is_empty]` param to /tags.json endpoint.
`search[hide_empty]=true` is deprecated in favor of `search[is_empty]=false`.
* On /pools, add option to show/hide deleted pools in search form.
* Fix the /forum_topics page putting `search[order]=sticky&limit=40` in
the URL when browsing past page 1.
Fix maintenance tasks failing to run in production. In production they
were losing the database connection and not re-establishing it, so they
couldn't queue jobs. `ApplicationRecord.verify!` will check if the
connection is lost and re-establish it if it is.
The database connection was being lost because in production we use a
Kubernetes service IP for the database IP, which is essentially a
virtual IP that maps to the real IP. This mapping is implemented with
IPVS[1][2], which has a default idle connection timeout of 5 minutes. If
the connection isn't used for more than 5 minutes, then it's closed.
Since maintenance only runs once an hour, the database connection would
be lost because it was idle for too long.
1: https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-ipvs
2: https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/
Remove the ability for users to lock ratings, note, and post statuses.
Historically the majority of locked posts were from 10+ years ago when
certain users habitually locked ratings and notes on every post they
touched for no reason. Nowadays most posts have been unlocked. Only a
handful of locked posts are left, none of which deserve to be locked.
The is_rating_locked, is_note_locked, and is_status_locked columns still
exist in the database, but aren't used.
Add methods to MediaFile to calculate the duration, frame count, and
frame rate of animated GIFs, PNGs, Ugoiras, and videos.
Some considerations:
* It's possible to have a GIF or PNG that's technically animated but
just has one frame. These are treated as non-animated images.
* It's possible to have an animated GIF that has an unspecified
frame rate. In this case we assume the frame rate is 10 FPS; this is
browser dependent and may not be correct.
* Animated GIFs, PNGs, and Ugoiras all support variable frame rates.
Technically, each frame has a separate delay, and the delays can be
different frame-to-frame. We report only the average frame rate.
* Getting the duration of an APNG is surprisingly hard. Most tools don't
have good support for APNGs since it's a rare and non-standardized
format. The best we can do is get the frame count using ExifTool and the
frame rate using ffprobe, then calculate the duration from that.
Add the following:
* Container name, machine name, worker id.
* Container uptime, puma uptime, worker uptime.
* Number of requests processed by current worker.
* ExifTool version.
Also change /status page to show information in tables instead of lists.
Break the hourly/daily/weekly/monthly maintenance tasks down into
individual delayed jobs. This way if one task fails, it won't prevent
other tasks from running. Also, jobs can be run in parallel, and can be
individually retried if they fail.
Move the metadata parsing code from MediaAsset to ExifTool::Metadata so
we can use it outside the context of a MediaAsset, in particular when
dealing with a MediaFile that hasn't been saved to disk yet.
Set `crossorigin="anonymous"` on the <img> tag for thumbnails. This
makes it so we don't send session cookies in requests for thumbnails.
Since images are served from cdn.donmai.us, and since session cookies
are set on *.donmai.us, session cookies are sent by default with every
thumbnail request.
Not sending cookies saves up to 1kb of overhead per thumbnail request
(in reality, it's less than this because of HTTP/2 multiplexing and
header compression).
Caused by d854bf6b. Banning an artist would deadlock because it was
performed in a transaction, which didn't work with the `parallel_each`
inside the "create an implication to banned_artist" step.
Fix a bug where where PNG images could be incorrectly detected as
exif-rotated. This would happen when a PNG contained the
IFD0:Orientation flag. It's technically possible for a PNG to contain
this flag, but it's ignored by libvips and by browsers.
post #3762340 (nsfw) is an example of a PNG like this.
The fix is to use `autorot` to let libvips apply the rotation instead of
trying to interpret the exif data ourselves. Note that libvips-8.9 has a
bug where it doesn't strip the orientation flag after applying
`autorot`, which leads to the image being incorrectly rotated a second
time when generating the thumbnail. Use libvips-8.11 instead.
Hourly pruning of expired uploads was failing because of nil deference
errors in `media_asset.destroy!`. There are various cases where an
upload doesn't have a media asset, for example when the source url
fails to download or when the upload is of an invalid filetype.
`IFD0:Orientation` is the orientation of the main image.
`IFD1:Orientation` is the orientation of the embedded thumbnail, if it
has one. Using `IFD1:Orientation` was incorrect here because some images
have a non-rotated main image but a rotated thumbnail. Post #1023563 is
an example.
Rotate the image based on the EXIF orientation flag when generating
thumbnails and samples.
Also fix the width and height to be calculated correctly for rotated
images. Vips gives us the unrotated width and height of the image; we
have to detect whether the image is rotated and swap the width and
height manually to correct them. For example, if an image with the
"Rotate 90 CW" flag is 100x500 before rotation, then after rotation it's
500x100. This should fix#4883 (Exif rotation breaks Javascript fit-to-window)
We also have to fix it so that regenerating a post updates the width and
height of the post, in the event that it's a rotated image.
Finally we set `image-orientation: from-image;` even though it's
probably not necessary.
