The problem was that we were stripping color profiles from thumbnails,
but we weren't setting `export_profile: "srgb"` to convert images to
sRGB first. This resulted in wrong colors for images with non-sRGB color
profiles, such as Adobe RGB.
The fix is to convert images to sRGB when possible, while leaving CMYK
and greyscale images alone. We leave CMYK images alone because we can't
convert CMYK to sRGB without losing color. We leave greyscale images
alone if they don't have a color profile, that way they stay as
one-channel greyscale (or two-channel greyscale, in case of alpha)
instead of being converted to three-channel sRGB. However, if a
greyscale image has a color profile, then we have to convert to sRGB,
otherwise the colors would be wrong when we strip the profile.
We also have to set the import profile, otherwise images with broken
embedded color profiles won't have a fallback profile and may get
incorrect colors. In this case we also have to be careful, because we
can't specify an sRGB fallback for greyscale or CMYK images.
Make the `order:random` metatag truly randomize the search. Add a
`random:N` metatag that returns up to N random posts, like what
`order:random` did before.
`order:random` now returns the entire search in random order. Before it
just returned a pageful of pseudorandom posts. This will be more
accurate for small searches, but slower for large searches. If
`order:random` times out, try `random:N` instead.
The `random:N` metatag returns up to N pseudorandom posts. This is
faster than `order:random` for large searches, but for small searches,
it may return less than N posts, and the randomness may be biased. Some
posts may be more likely than others to appear. N must be between 0 and
200.
Also, `/posts?tags=touhou&random=1` now redirects to `/posts?tags=touhou+random:N`.
Before the `random=1` param acted like a free `order:random` tag; now it
redirects to a `random:N` search, so it counts against your tag limit.
Allow admins to remove votes on posts. This is for fixing vote abuse.
Votes can be removed by going to the vote list on the /post_votes page,
or by clicking on a post's score, then using the "Remove" option in the
"..." dropdown menu next to the vote.
Votes are soft-deleted - they're marked as deleted in the database, but
not fully deleted. Removed votes are only visible to admins, not to
regular users. When a vote is removed by an admin, it leaves a mod
action.
Technically it's possible to undelete votes, but there's no UI for it.
* Add a gap between thumbnails on mobile.
* Adjust CSS for scores and vote buttons.
* Include "Private favorites" as an incentive on the user upgrade page.
* Fix vote buttons not being visible beneath thumbnails on mobile.
* Fix the "Show scores" link not preserving the current page number.
* Fix vote buttons being unintentionally enabled for all thumbnails by default.
* Fix banned and restricted users being able to favorite posts by
tagging them with `fav:self`.
* Fix search engines being able to crawl /posts?view=score pages.
* Fix broken tests.
Changes:
* Make it so you can click or hover over a post's favorite count to see
the list of public favorites.
* Remove the "Show »" button next to the favorite count.
* Make the favorites list visible to all users. Before favorites were
only visible to Gold users.
* Make the /favorites page show the list of all public favorites,
instead of redirecting to the current user's favorites.
* Add /posts/:id/favorites endpoint.
* Add /users/:id/favorites endpoint.
This is for several reasons:
* To make viewing favorites work the same way as viewing upvotes.
* To make posts load faster for Gold users. Before, we loaded all the
favorites when viewing a post, even when the user didn't look at them.
This made pageloads slower for posts that had hundreds or thousands of
favorites. Now we only load the favlist if the user hovers over the favcount.
* To make the favorite list visible to all users. Before, it wasn't
visible to non-Gold users, because of the performance issue listed above.
* To make it more obvious that favorites are public by default. Before,
since regular users could only see the favcount, they may have
mistakenly believed other users couldn't see their favorites.
Make private favgroups a Gold-only option. This is for consistency with
private favorites and upvotes being Gold-only options.
Existing Members with private favgroups are allowed to keep them, as
long as they don't disable privacy. If they disable it, then they can't
re-enable it again without upgrading to Gold first.
Make private favorites and upvotes a Gold-only account option.
Existing Members with private favorites enabled are allowed to keep it
enabled, as long as they don't disable it. If they disable it, then they
can't re-enable it again without upgrading to Gold first.
This is a Gold-only option to prevent uploaders from creating multiple
accounts to upvote their own posts. If private upvotes were allowed for
Members, then it would be too easy to use fake accounts and private
upvotes to upvote your own posts.
* Allow Member-level users to vote.
* Don't allow Banned or Restricted users to create favorites any more.
Banned and Restricted users aren't allowed to upvote or favorite any
more to prevent sockpuppet accounts from upvoting even after they're
banned.
Make upvotes public the same way favorites are public:
* Rename the "Private favorites" account setting to "Private favorites and upvotes".
* Make upvotes public, unless the user has private upvotes enabled. Note
that private upvotes are still visible to admins. Downvotes are still
hidden to everyone except for admins.
* Make https://danbooru.donmai.us/post_votes visible to all users. This
page shows all public upvotes. Private upvotes and downvotes are only
visible on the page to admins and to the voter themselves.
* Make votes searchable with the `upvote:username` and `downvote:username`
metatags. These already existed before, but they were only usable by
admins and by people searching for their own votes.
Upvotes are public to discourage users from upvoting with multiple
accounts. Upvote abuse is obvious to everyone when upvotes are public.
The other reason is to make upvotes consistent with favorites, which are
already public.
Add `upvotes:N`, `downvotes:N`, `order:upvotes`, `order:downvotes`,
`order:upvotes_asc`, `order:downvotes_asc` metatags.
In the API, the field is called up_score / down_score. Here it's called
`upvotes` and `downvotes` because this should be easier to understand
for end users.
Note that internally, `down_score` is negative. A post that matches
`downvotes:>5` will have down_score < -5 internally.
Fix certain artist commentaries for foundation.app containing scrambled
characters. Apparently caused by the Nokogiri HTML5 parser not handling
UTF-8 input correctly when the encoding isn't explicitly set to UTF-8.
Bug: If a media asset got stuck in the 'processing' state during upload,
then it would stay stuck forever and the file couldn't be uploaded again
later.
Fix: Mark stuck assets as failed before raising the "Upload failed"
error. Once the asset is marked as failed, it can be uploaded again
later. Also, only wait for assets to finish processing if they were
uploaded less than 5 minutes ago. If a processing asset is more than 5
minutes old, consider it stuck and mark it as failed immediately.
Assets getting stuck in the processing state is a 'this should never
happen' error. Normally if any kind of exception is raised while
uploading the asset, the asset will be set to the 'failed' state. The
only way an asset can get stuck is if it fails and the exception handler
doesn't run, or the exception handler itself fails. This might happen if
the process is unexpectedly killed, or possibly if the HTTP request
times out and a TimeoutError is raised at an inopportune time. See below
for discussion of issues with Timeout.
[1]: https://vaneyckt.io/posts/the_disaster_that_is_rubys_timeout_method/
[2]: https://jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying/
[3]: https://adamhooper.medium.com/in-ruby-dont-use-timeout-77d9d4e5a001
[4]: https://ruby-doc.org/core-3.0.2/Thread.html#method-c-handle_interrupt-label-Guarding+from+Timeout-3A-3AError
Change the rules for automatically retiring aliases and implications:
* Retire aliases to tags that are empty, or that are for a general or
artist tag that hasn't received any new posts in the last two years.
* Retire implications from tags that are empty.
* Don't retire aliases or implications for character, copyright, or
meta tags any more, unless the tags are empty.
Make it possible to reapprove failed BURs that removed aliases or
implications.
Before if a BUR failed midway through, and we tried to reapprove it,
then it would fail when it got to a `remove alias` line because the
alias had already been removed. Now we keep going if we try to remove an
alias or implication that has already been removed.
Change the /bulk_update_requests page to show the newest BURs first
instead of pending BURs first. This is more consistent with index pages
on the rest of the site, which normally default to newest first. Fixes
an issue where failed BURs would be shown first forever unless they were
manually approved or rejected.
Fix the video duration not being shown on thumbnails on the
https://danbooru.donmai.us/comments page.
BUG: this introduces duplicate HTML ids on the comments page. Post
thumbnails and post comment containers both have the same html ID.
* Change `age:` metatag to require time units. This means e.g.
`age:<600` no longer works; instead you have to say `age:<600sec`.
* Allow time units in the `age:` metatag to be abbreviated as long as
they're unambiguous. This means `age:<60sec`, `age:<5min`, and
`age:<5mon` now work, in addition to `age:<60s` and `age:<60seconds`.
* Allow the `ratio:` metatag to be written like `ratio:16/9` in addition
to `ratio:16:9`.
* Fix invalid date searches like `date:foo` or `date:05-15-2021`
to return nothing instead of raising an "undefined method
'beginning_of_day' for nil" exception. (`date:05-15-2021` is invalid
because it's parsed as DD-MM-YYYY).
* Fix invalid searches like `score:foo`, `ratio:foo`, and `mpixels:foo`
to return nothing instead of being treated like `score:0`, `ratio:0`,
`mpixels:0`.
* Fix `age:<60m` to return nothing instead of silently being treated
like `age:<60seconds`.
* Fix `age:foo` to return nothing instead of silently being treated like
`age:0d` (return all uploads from today).
Fixes#4389.
Refactor StorageManager to remove all image URL generation code. Instead
the image URL generation code lives in MediaAsset.
Now StorageManager is only concerned with how to read and write files to
remote storage backends like S3 or SFTP, not with how image URLs should
be generated. This way the file storage code isn't tightly coupled to
posts, so it can be used to store any kind of file, not just images
belonging to posts.
Fix it so that when a post is expunged, the media asset is also marked
as expunged. This way the files will be deleted, but the media asset
will still remain as a record of what was expunged. The media asset will
have the md5, width, height, file ext, and file size of the deleted file.
Show the length of videos and animated posts in the thumbnail. The
length is shown the top left corner in MM:SS format. This replaces the
play button icon.
Show a speaker icon instead of a music note icon for posts with sound.
Doing this requires doing `.includes(:media_asset)` in a bunch of
places to avoid N+1 queries when we access the post's duration.
Don't delete replaced files after 30 days. There are only about 30k
replacements in total, so the cost of keeping replaced files is
negligible. It was also wrong because the media asset wasn't destroyed
too, so there were active media assets with missing files.
Add a md5 uniqueness constraint on media assets to prevent duplicate
assets from being created. This way we can guarantee that there is one
active media asset per uploaded file.
Also make it so that if two people are uploading the same file at the
same time, the file is processed only once.
* Make it so replacing a post doesn't generate a dummy upload as a side effect.
* Make it so you can't replace a post with itself (the post should be regenerated instead).
* Refactor uploads and replacements to save the ugoira frame data when
the MediaAsset is created, not when the post is created. This way it's
possible to view the ugoira before the post is created.
* Make `download_file!` in the Pixiv source strategy return a MediaFile
with the ugoira frame data already attached to it, instead of returning it
in the `data` field then passing it around separately in the `context`
field of the upload.
Move more of the file-handling logic from UploadService and
StorageManager into MediaAsset. This is part of refactoring posts and
uploads to allow multiple images per post.
Fix how the duration of videos and animated GIFs / PNGs is calculated.
If we can't determine the duration from the file metadata, then play the
entire video or animation back using FFmpeg and scrape the duration and
frame count.
This is necessary for things like WebM files where the duration metadata
is optional, or animated GIFs and PNGs that don't have a duration field
in the metadata, only a frame count and a sequence of frame delays.
