Bug: If a media asset got stuck in the 'processing' state during upload,
then it would stay stuck forever and the file couldn't be uploaded again
later.
Fix: Mark stuck assets as failed before raising the "Upload failed"
error. Once the asset is marked as failed, it can be uploaded again
later. Also, only wait for assets to finish processing if they were
uploaded less than 5 minutes ago. If a processing asset is more than 5
minutes old, consider it stuck and mark it as failed immediately.
Assets getting stuck in the processing state is a 'this should never
happen' error. Normally if any kind of exception is raised while
uploading the asset, the asset will be set to the 'failed' state. The
only way an asset can get stuck is if it fails and the exception handler
doesn't run, or the exception handler itself fails. This might happen if
the process is unexpectedly killed, or possibly if the HTTP request
times out and a TimeoutError is raised at an inopportune time. See below
for discussion of issues with Timeout.
[1]: https://vaneyckt.io/posts/the_disaster_that_is_rubys_timeout_method/
[2]: https://jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying/
[3]: https://adamhooper.medium.com/in-ruby-dont-use-timeout-77d9d4e5a001
[4]: https://ruby-doc.org/core-3.0.2/Thread.html#method-c-handle_interrupt-label-Guarding+from+Timeout-3A-3AError
Change the rules for automatically retiring aliases and implications:
* Retire aliases to tags that are empty, or that are for a general or
artist tag that hasn't received any new posts in the last two years.
* Retire implications from tags that are empty.
* Don't retire aliases or implications for character, copyright, or
meta tags any more, unless the tags are empty.
Make it possible to reapprove failed BURs that removed aliases or
implications.
Before if a BUR failed midway through, and we tried to reapprove it,
then it would fail when it got to a `remove alias` line because the
alias had already been removed. Now we keep going if we try to remove an
alias or implication that has already been removed.
Change the /bulk_update_requests page to show the newest BURs first
instead of pending BURs first. This is more consistent with index pages
on the rest of the site, which normally default to newest first. Fixes
an issue where failed BURs would be shown first forever unless they were
manually approved or rejected.
Fix the video duration not being shown on thumbnails on the
https://danbooru.donmai.us/comments page.
BUG: this introduces duplicate HTML ids on the comments page. Post
thumbnails and post comment containers both have the same html ID.
* Change `age:` metatag to require time units. This means e.g.
`age:<600` no longer works; instead you have to say `age:<600sec`.
* Allow time units in the `age:` metatag to be abbreviated as long as
they're unambiguous. This means `age:<60sec`, `age:<5min`, and
`age:<5mon` now work, in addition to `age:<60s` and `age:<60seconds`.
* Allow the `ratio:` metatag to be written like `ratio:16/9` in addition
to `ratio:16:9`.
* Fix invalid date searches like `date:foo` or `date:05-15-2021`
to return nothing instead of raising an "undefined method
'beginning_of_day' for nil" exception. (`date:05-15-2021` is invalid
because it's parsed as DD-MM-YYYY).
* Fix invalid searches like `score:foo`, `ratio:foo`, and `mpixels:foo`
to return nothing instead of being treated like `score:0`, `ratio:0`,
`mpixels:0`.
* Fix `age:<60m` to return nothing instead of silently being treated
like `age:<60seconds`.
* Fix `age:foo` to return nothing instead of silently being treated like
`age:0d` (return all uploads from today).
Fixes#4389.
Refactor StorageManager to remove all image URL generation code. Instead
the image URL generation code lives in MediaAsset.
Now StorageManager is only concerned with how to read and write files to
remote storage backends like S3 or SFTP, not with how image URLs should
be generated. This way the file storage code isn't tightly coupled to
posts, so it can be used to store any kind of file, not just images
belonging to posts.
Fix it so that when a post is expunged, the media asset is also marked
as expunged. This way the files will be deleted, but the media asset
will still remain as a record of what was expunged. The media asset will
have the md5, width, height, file ext, and file size of the deleted file.
Show the length of videos and animated posts in the thumbnail. The
length is shown the top left corner in MM:SS format. This replaces the
play button icon.
Show a speaker icon instead of a music note icon for posts with sound.
Doing this requires doing `.includes(:media_asset)` in a bunch of
places to avoid N+1 queries when we access the post's duration.
Don't delete replaced files after 30 days. There are only about 30k
replacements in total, so the cost of keeping replaced files is
negligible. It was also wrong because the media asset wasn't destroyed
too, so there were active media assets with missing files.
Add a md5 uniqueness constraint on media assets to prevent duplicate
assets from being created. This way we can guarantee that there is one
active media asset per uploaded file.
Also make it so that if two people are uploading the same file at the
same time, the file is processed only once.
* Make it so replacing a post doesn't generate a dummy upload as a side effect.
* Make it so you can't replace a post with itself (the post should be regenerated instead).
* Refactor uploads and replacements to save the ugoira frame data when
the MediaAsset is created, not when the post is created. This way it's
possible to view the ugoira before the post is created.
* Make `download_file!` in the Pixiv source strategy return a MediaFile
with the ugoira frame data already attached to it, instead of returning it
in the `data` field then passing it around separately in the `context`
field of the upload.
Move more of the file-handling logic from UploadService and
StorageManager into MediaAsset. This is part of refactoring posts and
uploads to allow multiple images per post.
Fix how the duration of videos and animated GIFs / PNGs is calculated.
If we can't determine the duration from the file metadata, then play the
entire video or animation back using FFmpeg and scrape the duration and
frame count.
This is necessary for things like WebM files where the duration metadata
is optional, or animated GIFs and PNGs that don't have a duration field
in the metadata, only a frame count and a sequence of frame delays.
When a search is performed, we cache the post count so we don't have to
calculate it again every time the user switches pages. However, if the
count times out, we didn't cache it before, causing us to do a slow
count on every page load. This usually happens on multi-tag searches
that return a lot of results, `1girl solo` for example.
This changes it so that the count is cached even when it times out. This
will speed up large multi-tag searches.
This also changes it so that the count is cached for a fixed 5 minutes.
Before it was variable based on the size of the count, but this probably
didn't make much difference.
Use the `string_to_array(tag_string, ' ')` index instead of the
`tag_index` for tag searches. The string_to_array index lets us treat
the tag_string as an array for searching purposes. This lets us get rid
of the tag_index column and the test_parser dependency in the future.
Stop updating the fav_string attribute on posts. The column still exists
on the table, but is no longer used or updated.
Like the pool_string in 7d503f08, the fav_string was used in the past to
facilitate `fav:X` searches. Posts had a hidden fav_string column that
contained a list of every user who favorited the post. These were
treated like fake hidden tags on the post so that a search for `fav:X`
was treated like a tag search.
The fav_string attribute has been unused for search purposes for a while
now. It was only kept because of technicalities that required
departitioning the favorites table first (340e1008e) before it could be
removed. Basically, removing favorites with `@favorite.destroy` was
slow because Rails always deletes object by ID, but we didn't have an
index on favorites.id, and we couldn't easily add one until the
favorites table was departitioned.
Fixes#4652. See https://github.com/danbooru/danbooru/issues/4652#issuecomment-754993802
for more discussion of issues caused by the fav_string (in short: write
amplification, post table bloat, and favorite inconsistency problems).
Optimize counting the number of posts returned by fav:<name> and
pool:<name> searches. Use cached counts to avoid slow count(*) queries
for users with lots of favorites.
Let all users have unlimited favorites. Formerly the limit was 10k
favorites for regular members, 20k for Gold, and unlimited for Platinum.
Limiting favorites doesn't make sense since upvotes are unlimited.
Stop using the pool_string attribute on posts:
* Stop updating it when adding or removing posts from pools.
* Stop returning pool_string in the /posts.json API.
* Stop including the `data-pools` attribute on thumbnails.
The pool_string attribute was used in the past to facilitate pool:X
searches. Posts had a hidden pool_string attribute that contained a list
of every pool the post belonged to. These pools were treated like fake
hidden tags on the post and a search for `pool:X` was treated like a tag
search.
The pool_string has no longer been used for this purpose for a long time
now, and was only maintained for API compatibility purposes. Getting rid
of it eliminates a bunch of legacy cruft relating to adding and removing
posts from pools.
If you need to see which pools a post belongs to, do this:
* https://danbooru.donmai.us/pools.json?search[post_ids_include_any]=318550
The `data-pools` attribute on thumbnails was used by some people to add
custom borders to pooled posts with custom CSS. This will no longer
work. This was already broken because it included things like collection
pools and deleted pools, which you probably didn't want. Use a
userscript to add this attribute back to thumbnails if you need it.
Fix certain animated PNGs returning NaN as the duration because the
frame rate was being reported as "0/0" by FFMpeg. This happens when the
animation has zero delay between frames. This is supposed to mean a PNG
with an infinitely fast frame rate, but in practice browsers limit it to
around 10FPS. The exact frame rate browsers will use is unknown and
implementation defined.
* Skip Nijie tests because they fail a lot due to Nijie rate limiting us.
* Skip ArtStation downloads tests because they sometimes return different file sizes.
* Fix random duplicate favgroup errors because favgroup names weren't random enough.
* On /pools, hide deleted pools by default in HTML responses. Don't
filter out deleted pools in API responses.
* API change: on /forum_topics, only hide deleted forum topics by
default for HTML responses, not for API responses. Explicitly do
https://danbooru.donmai.us/forum_topics.json?search[is_deleted]=false
to filter out deleted topics.
* API change: on /tags, only hide empty tags by default for HTML
responses, not for API responses. Explicitly do
https://danbooru.donmai.us/tags.json?search[is_empty]=false to filter
out empty tags.
* API change: on /pools, default to 20 posts per page for API responses,
not 40.
* API change: add `search[is_empty]` param to /tags.json endpoint.
`search[hide_empty]=true` is deprecated in favor of `search[is_empty]=false`.
* On /pools, add option to show/hide deleted pools in search form.
* Fix the /forum_topics page putting `search[order]=sticky&limit=40` in
the URL when browsing past page 1.
