Add support for uploading posts from Gelbooru. Note that the translated
tags will include both the Gelbooru tags and the tags from the Gelbooru
post's source. The commentary and artist information will also be taken
from the Gelbooru post's source. The source of the Danbooru post however
will be left as the Gelbooru post itself, not as the Gelbooru post's source.
Remove the last remaining uses of the PixivUgoiraFrameData model. As of
32bfb8407, Ugoira frame data is now stored in the MediaMetadata model,
under the `Ugoira:FrameDelays` EXIF field.
The pixiv_ugoira_frame_data table still exists, but it can be removed
after this commit is deployed.
Fixes#5264: Error when replacing with ugoira.
Automatically add the AI-generated tag to posts that have the
`PNG:Software=NovelAI` EXIF attribute.
This is not foolproof because this metadata may get removed if an
AI-generated post is resaved or uploaded to a site that strips EXIF
metadata. It also only works for NovelAI. Currently it detects 29 out of
177 AI-generated uploads on Danbooru.
Switch the Ugoira Javascript player to using frame delays from the
MediaMetadata model instead of from the PixivUgoiraFrameData model.
After this is deployed it will be safe to remove the PixivUgoiraFrameData
model.
Store Ugoira frame delays in the MediaMetadata model as a fake EXIF
field instead of in the PixivUgoiraFrameData model. This way we can get
rid of the PixivUgoiraFrameData model completely. This is a step towards
fixing #5264.
Remove various fallbacks and workarounds from the Ugoira Javascript
player for legacy browser issues.
* Remove fallbacks for URL, Blob, and ArrayBuffer (these have been
supported in most browsers for years).
* Remove workaround for iOS <6 not supporting Blob URLs.
* Remove workaround for a bug in Safari's handling of the Range HTTP
header. This was fixed in 2015.
* Remove support for playing a list of image URLs instead of playing
images from the .zip file (this was never used).
* Remove support for limiting loadahead of images (it was hardcoded to
always use unlimited loadahead anyways).
The frame data for Ugoira files is stored like this:
[{"file"=>"000000.jpg", "delay"=>65},
{"file"=>"000001.jpg", "delay"=>65},
{"file"=>"000002.jpg", "delay"=>65},
{"file"=>"000003.jpg", "delay"=>65},
{"file"=>"000004.jpg", "delay"=>65},
{"file"=>"000005.jpg", "delay"=>65},
{"file"=>"000006.jpg", "delay"=>65},
{"file"=>"000007.jpg", "delay"=>65},
{"file"=>"000008.jpg", "delay"=>65},
{"file"=>"000009.jpg", "delay"=>65},
{"file"=>"000010.jpg", "delay"=>65}]
This is stored in the pixiv_ugoira_frame_data table in YAML format. This
is a problem because a) we only need the frame delays to play the Ugoira,
not the filenames, and b) storing the data in YAML format is a security
issue that's blocking the upgrade to Rails 7.0.4 (see [1]).
This commit changes the Ugoira Javascript player so that it only uses
the list of frame delays, not the filenames, to play the Ugoira. This
paves the way for storing the frame delays as a simple integer array
instead as a serialized YAML object.
This assumes that the images in a Ugoira zip file are stored in the same
order they should be played back in. This was confirmed by checking every
zip file and verifying that files are actually stored in filename order.
[1]: https://discuss.rubyonrails.org/t/cve-2022-32224-possible-rce-escalation-bug-with-serialized-columns-in-active-record/81017
Whenever the email address normalization procedure changes, the
`normalized_address` column of the email address table must be updated.
This is normally when the list of canonical domain mappings changes.
Renormalizing addresses may also require deleting duplicates.
In the past it was possible for users to create multiple accounts with
the same email address. We had about 9000 such accounts. This removes
the email address from these accounts.
When multiple accounts have the same email address, the account that
visited the site last gets to keep the address.
Allow moderators to forcibly change the username of other users. This is
so mods can change abusive or invalid usernames.
* A mod can only change the username of Builder-level users and below.
* The user can't change their own name again until one week has passed.
* A modaction is logged when a mod changes a user's name.
* A dmail is sent to the user notifying them of the change.
* The dmail does not send the user an email notification. This is so we
don't spam users if their name is changed after they're banned, or if
they haven't visited the site in a long time.
The rename button is on the user's profile page, and when you hover over
the user's name and open the "..." menu.
* Fixed a bug where manga posts with a single tag would raise an error
* Fixed a bug where dic.nicovideo.jp/oekaki posts weren't uploadable due
to SSL issues
* Added support for more manga corner cases
Fix not being able to use the full set of search operators on polymorphic `model_id` and
`model_type` attributes. Before things like `search[model_type]=Post` worked, but
`search[model_type_not_eq]=Post` or other `model_type_*` operators didn't.
Log the following information in email headers:
* X-Danbooru-User: the user's name and ID.
* X-Danbooru-IP: the user's IP.
* X-Danbooru-Session: the users' session ID.
* X-Danbooru-URL: the page that triggered the email.
* X-Danbooru-Job-Id: the ID of the background job that sent the email.
* X-Danbooru-Enqueued-At: when the email was queued as a background job.
* X-Danbooru-Dmail: for Dmail notifications, the link to the Dmail.
* X-Request-Id: the request ID of the HTTP request that triggered the email.
Also make it so we log an event in the APM when we send an email.
Fix it so that emails are (hopefully) able to show the one-click
unsubscribe button in Gmail and other mail providers that support the
List-Unsubscribe header. This way users can unsubscribe instead of
marking emails as spam.
* Add the List-Unsubscribe-Post header.
* Fix the disable email notifications endpoint to support POST as well as DELETE requests.
* Fix the disable email notifications endpoint to disable XSRF protection (we don't need
users to be logged in because we use a signed URL instead).
https://www.rfc-editor.org/rfc/rfc8058#section-3.1https://www.rfc-editor.org/rfc/rfc8058#section-8.1
Fix a bug where the modqueue didn't show the correct page count when
filtering by tag. Before we showed the total number of posts in the
modqueue, instead of the actual number of posts for your search. This
was for speed reasons, but with recent fixes it should be less of a
problem to count the actual number of pages for the search.
Fix a bug where searching for a negated tag inside the modqueue would show
active posts.
The bug was that in a search like this:
Post.in_modqueue.user_tag_match("-solo")
The `in_modqueue` condition would get sucked inside the tag search and negated
when we tried to apply the negation operator to the "solo" tag. So effectively
the `in_modqueue` condition would get negated and we would end up searching for
everything not in the modqueue.
Fix the order dropdown box on the modqueue page not working when filtering by tag.
This happened because when you do a tag search, the default order is set to `ORDER BY posts.id DESC`.
When you applied another order with the dropdown box, the new order would be tacked on to the old
ordering as a tiebreaker instead of replacing it, producing e.g. `ORDER BY posts.id DESC, queued_at DESC`
instead of `ORDER BY queued_at DESC`. The default order would always win because `posts.id` is
unique and doesn't have ties.
The fix is to have orders always override the previous order instead of adding to it.
Note that now if you use an `order:`, `ordfav:`, or `ordpool:` metatag in the search box on the
modqueue page, they will always be ignored in favor of the dropdown box.
Don't cache the page count for status:unmoderated, status:modqueue, status:pending, status:flagged,
or status:appealed searches. Before the page count was cached for 5 minutes, which could quickly
become out of date as a user went through the modqueue. After bd21b4a86, they're now fast enough
that we no longer need to cache them.
* Optimize status:modqueue and status:unmoderated searches. This brings them down from
taking 500ms-1000ms per search to ~5ms.
* Change status:unmoderated so that it only filters out the user's disapproved posts, not
the user's own uploads or past approvals. Now it's equivalent to `status:modqueue -disapproved:evazion`,
whereas before it was equivalent to `status:modqueue -disapproved:evazion -approver:evazion -user:evazion`.
Filtering out the user's own uploads and approvals was slow and usually unnecessary,
since for most users it's rare for their own uploads or approvals to reenter the modqueue.
Before status:modqueue did this:
SELECT * FROM posts WHERE is_pending = TRUE OR is_flagged = TRUE OR (is_deleted = TRUE AND id IN (SELECT post_id FROM post_appeals WHERE status = 0))
Now we do this:
SELECT * FROM posts WHERE id IN (SELECT id FROM posts WHERE is_pending = TRUE UNION ALL SELECT id FROM posts WHERE is_flagged = TRUE UNION ALL SELECT id FROM posts WHERE id IN (SELECT post_id FROM post_appeals WHERE status = 0))
Postgres had a bad time with the "pending or flagged or has a pending appeal" clause because
it didn't know that posts can only be in one state at a time, so it overestimated how many
posts would be returned and chose a seq scan. Replacing the OR with a UNION avoids this.
* Add header with the Danbooru name and logo.
* Add footer with links to the site, the privacy policy, and the contact page.
* Add "You received this email because of X" messages to remind users why
they received the email.
* Add basic CSS to make the design match the site.
Support searches like the following:
* score:<0,>100 (equivalent to `score:<0 or score:>100`)
* score:5,10..15,>20 (equivalent to `score:5 or score:10..15 or score:>20`)
* id:5,10..15,20..25,30 (equivalent to `id:5 or id:10..15 or id:20..25 or id:30`)
This also works inside the `search[id]` URL parameter, and inside other numeric
URL search parameters.
Factor out the code that parses metatag values (e.g. `score:>5`) and
search URL params (e.g. `search[score]=>5`) into a RangeParser class.
Also fix a bug where, if the `search[order]=custom` param was used
without a `search[id]` param, an exception would be raised. Fix another
bug where if an invalid `search[id]` was provided, then the custom order
would be ignored and the search would be returned with the default order
instead. Now if you use `search[order]=custom` without a valid
`search[id]` param, the search will return no results.
Fix the approver:, parent:, and pixiv: metatags not working correctly when negated:
* Fix -approver:<name> not including posts that don't have an approver (the approver_id is NULL)
* Fix -parent:<id> not including posts that don't have a parent (the parent_id is NULL)
* Fix -pixiv:<id> not including posts that aren't from Pixiv (the pixiv_id is NULL)
The problem lies how the equality operator is negated when the column contains NULL values;
`approver_id != 52664` doesn't match posts where the `approver_id` is NULL.
The search `approver:evazion` boils down to:
# Post.where(approver_id: 52664).to_sql
SELECT * FROM posts WHERE approver_id = 52664;
When that is negated with `-approver:evazion`, it becomes:
# Post.where(approver_id: 52664).invert_where.to_sql
SELECT * FROM posts WHERE approver_id != 52664;
But in SQL, `approver_id != 52664` doesn't match when the approver_id IS NULL, so the search doesn't
include posts without an approver.
We could use `a IS NOT DISTINCT FROM b` instead of `a = b`:
# Post.where(Post.arel_table[:approver_id].is_not_distinct_from(52664)).to_sql
SELECT * FROM posts WHERE approver_id IS NOT DISTINCT FROM 52664;
This way when it's inverted it becomes `IS DISTINCT FROM`:
# Post.where(Post.arel_table[:approver_id].is_not_distinct_from(52664)).invert_where.to_sql
SELECT * FROM posts WHERE approver_id IS NOT DISTINCT FROM 52664;
`approver_id IS DISTINCT FROM 52664` is like `approver_id != 52664`, except it matches when
approver_id is NULL [1].
This works correctly, however the problem is that `IS NOT DISTINCT FROM` can't use indexes because
of a long-standing Postgres limitation [2]. This makes searches too slow. So instead we do this:
# Post.where(approver_id: 52664).where.not(approver_id: nil).to_sql
SELECT * FROM posts WHERE approver_id = 52664 AND approver_id IS NOT NULL;
That way when negated it becomes:
# Post.where(approver_id: 52664).where.not(approver_id: nil).invert_where.to_sql
SELECT * FROM posts WHERE approver_id != 52664 OR approver_id IS NULL;
Which is the correct behavior.
[1] https://modern-sql.com/feature/is-distinct-from
[2] https://www.postgresql.org/message-id/6FC83909-5DB1-420F-9191-DBE533A3CEDE@excoventures.com
