Add support for uploading posts from Gelbooru. Note that the translated
tags will include both the Gelbooru tags and the tags from the Gelbooru
post's source. The commentary and artist information will also be taken
from the Gelbooru post's source. The source of the Danbooru post however
will be left as the Gelbooru post itself, not as the Gelbooru post's source.
Remove the last remaining uses of the PixivUgoiraFrameData model. As of
32bfb8407, Ugoira frame data is now stored in the MediaMetadata model,
under the `Ugoira:FrameDelays` EXIF field.
The pixiv_ugoira_frame_data table still exists, but it can be removed
after this commit is deployed.
Fixes#5264: Error when replacing with ugoira.
Automatically add the AI-generated tag to posts that have the
`PNG:Software=NovelAI` EXIF attribute.
This is not foolproof because this metadata may get removed if an
AI-generated post is resaved or uploaded to a site that strips EXIF
metadata. It also only works for NovelAI. Currently it detects 29 out of
177 AI-generated uploads on Danbooru.
Switch the Ugoira Javascript player to using frame delays from the
MediaMetadata model instead of from the PixivUgoiraFrameData model.
After this is deployed it will be safe to remove the PixivUgoiraFrameData
model.
Store Ugoira frame delays in the MediaMetadata model as a fake EXIF
field instead of in the PixivUgoiraFrameData model. This way we can get
rid of the PixivUgoiraFrameData model completely. This is a step towards
fixing #5264.
Remove various fallbacks and workarounds from the Ugoira Javascript
player for legacy browser issues.
* Remove fallbacks for URL, Blob, and ArrayBuffer (these have been
supported in most browsers for years).
* Remove workaround for iOS <6 not supporting Blob URLs.
* Remove workaround for a bug in Safari's handling of the Range HTTP
header. This was fixed in 2015.
* Remove support for playing a list of image URLs instead of playing
images from the .zip file (this was never used).
* Remove support for limiting loadahead of images (it was hardcoded to
always use unlimited loadahead anyways).
The frame data for Ugoira files is stored like this:
[{"file"=>"000000.jpg", "delay"=>65},
{"file"=>"000001.jpg", "delay"=>65},
{"file"=>"000002.jpg", "delay"=>65},
{"file"=>"000003.jpg", "delay"=>65},
{"file"=>"000004.jpg", "delay"=>65},
{"file"=>"000005.jpg", "delay"=>65},
{"file"=>"000006.jpg", "delay"=>65},
{"file"=>"000007.jpg", "delay"=>65},
{"file"=>"000008.jpg", "delay"=>65},
{"file"=>"000009.jpg", "delay"=>65},
{"file"=>"000010.jpg", "delay"=>65}]
This is stored in the pixiv_ugoira_frame_data table in YAML format. This
is a problem because a) we only need the frame delays to play the Ugoira,
not the filenames, and b) storing the data in YAML format is a security
issue that's blocking the upgrade to Rails 7.0.4 (see [1]).
This commit changes the Ugoira Javascript player so that it only uses
the list of frame delays, not the filenames, to play the Ugoira. This
paves the way for storing the frame delays as a simple integer array
instead as a serialized YAML object.
This assumes that the images in a Ugoira zip file are stored in the same
order they should be played back in. This was confirmed by checking every
zip file and verifying that files are actually stored in filename order.
[1]: https://discuss.rubyonrails.org/t/cve-2022-32224-possible-rce-escalation-bug-with-serialized-columns-in-active-record/81017
Whenever the email address normalization procedure changes, the
`normalized_address` column of the email address table must be updated.
This is normally when the list of canonical domain mappings changes.
Renormalizing addresses may also require deleting duplicates.
In the past it was possible for users to create multiple accounts with
the same email address. We had about 9000 such accounts. This removes
the email address from these accounts.
When multiple accounts have the same email address, the account that
visited the site last gets to keep the address.
Add a fix script that fixes invalid email addresses if they can be
fixed, otherwise they're deleted.
For a long time we didn't have any email validation, so we ended up with
a lot of invalid email addresses containing typos or other random garbage.
This tries to fix the most common typos when possible, otherwise the
email address is deleted.
In many cases the user created two accounts, one with a typo in the
email and one with the correct email. In these cases we can't fix the
invalid email, so we just delete it.
Allow moderators to forcibly change the username of other users. This is
so mods can change abusive or invalid usernames.
* A mod can only change the username of Builder-level users and below.
* The user can't change their own name again until one week has passed.
* A modaction is logged when a mod changes a user's name.
* A dmail is sent to the user notifying them of the change.
* The dmail does not send the user an email notification. This is so we
don't spam users if their name is changed after they're banned, or if
they haven't visited the site in a long time.
The rename button is on the user's profile page, and when you hover over
the user's name and open the "..." menu.
Fix not being able to use the full set of search operators on polymorphic `model_id` and
`model_type` attributes. Before things like `search[model_type]=Post` worked, but
`search[model_type_not_eq]=Post` or other `model_type_*` operators didn't.
Log the following information in email headers:
* X-Danbooru-User: the user's name and ID.
* X-Danbooru-IP: the user's IP.
* X-Danbooru-Session: the users' session ID.
* X-Danbooru-URL: the page that triggered the email.
* X-Danbooru-Job-Id: the ID of the background job that sent the email.
* X-Danbooru-Enqueued-At: when the email was queued as a background job.
* X-Danbooru-Dmail: for Dmail notifications, the link to the Dmail.
* X-Request-Id: the request ID of the HTTP request that triggered the email.
Also make it so we log an event in the APM when we send an email.
Fix it so that emails are (hopefully) able to show the one-click
unsubscribe button in Gmail and other mail providers that support the
List-Unsubscribe header. This way users can unsubscribe instead of
marking emails as spam.
* Add the List-Unsubscribe-Post header.
* Fix the disable email notifications endpoint to support POST as well as DELETE requests.
* Fix the disable email notifications endpoint to disable XSRF protection (we don't need
users to be logged in because we use a signed URL instead).
https://www.rfc-editor.org/rfc/rfc8058#section-3.1https://www.rfc-editor.org/rfc/rfc8058#section-8.1
Fix a bug where the modqueue didn't show the correct page count when
filtering by tag. Before we showed the total number of posts in the
modqueue, instead of the actual number of posts for your search. This
was for speed reasons, but with recent fixes it should be less of a
problem to count the actual number of pages for the search.
Fix a bug where searching for a negated tag inside the modqueue would show
active posts.
The bug was that in a search like this:
Post.in_modqueue.user_tag_match("-solo")
The `in_modqueue` condition would get sucked inside the tag search and negated
when we tried to apply the negation operator to the "solo" tag. So effectively
the `in_modqueue` condition would get negated and we would end up searching for
everything not in the modqueue.
Fix the order dropdown box on the modqueue page not working when filtering by tag.
This happened because when you do a tag search, the default order is set to `ORDER BY posts.id DESC`.
When you applied another order with the dropdown box, the new order would be tacked on to the old
ordering as a tiebreaker instead of replacing it, producing e.g. `ORDER BY posts.id DESC, queued_at DESC`
instead of `ORDER BY queued_at DESC`. The default order would always win because `posts.id` is
unique and doesn't have ties.
The fix is to have orders always override the previous order instead of adding to it.
Note that now if you use an `order:`, `ordfav:`, or `ordpool:` metatag in the search box on the
modqueue page, they will always be ignored in favor of the dropdown box.
Don't cache the page count for status:unmoderated, status:modqueue, status:pending, status:flagged,
or status:appealed searches. Before the page count was cached for 5 minutes, which could quickly
become out of date as a user went through the modqueue. After bd21b4a86, they're now fast enough
that we no longer need to cache them.
* Optimize status:modqueue and status:unmoderated searches. This brings them down from
taking 500ms-1000ms per search to ~5ms.
* Change status:unmoderated so that it only filters out the user's disapproved posts, not
the user's own uploads or past approvals. Now it's equivalent to `status:modqueue -disapproved:evazion`,
whereas before it was equivalent to `status:modqueue -disapproved:evazion -approver:evazion -user:evazion`.
Filtering out the user's own uploads and approvals was slow and usually unnecessary,
since for most users it's rare for their own uploads or approvals to reenter the modqueue.
Before status:modqueue did this:
SELECT * FROM posts WHERE is_pending = TRUE OR is_flagged = TRUE OR (is_deleted = TRUE AND id IN (SELECT post_id FROM post_appeals WHERE status = 0))
Now we do this:
SELECT * FROM posts WHERE id IN (SELECT id FROM posts WHERE is_pending = TRUE UNION ALL SELECT id FROM posts WHERE is_flagged = TRUE UNION ALL SELECT id FROM posts WHERE id IN (SELECT post_id FROM post_appeals WHERE status = 0))
Postgres had a bad time with the "pending or flagged or has a pending appeal" clause because
it didn't know that posts can only be in one state at a time, so it overestimated how many
posts would be returned and chose a seq scan. Replacing the OR with a UNION avoids this.
* Add header with the Danbooru name and logo.
* Add footer with links to the site, the privacy policy, and the contact page.
* Add "You received this email because of X" messages to remind users why
they received the email.
* Add basic CSS to make the design match the site.
Support searches like the following:
* score:<0,>100 (equivalent to `score:<0 or score:>100`)
* score:5,10..15,>20 (equivalent to `score:5 or score:10..15 or score:>20`)
* id:5,10..15,20..25,30 (equivalent to `id:5 or id:10..15 or id:20..25 or id:30`)
This also works inside the `search[id]` URL parameter, and inside other numeric
URL search parameters.
Factor out the code that parses metatag values (e.g. `score:>5`) and
search URL params (e.g. `search[score]=>5`) into a RangeParser class.
Also fix a bug where, if the `search[order]=custom` param was used
without a `search[id]` param, an exception would be raised. Fix another
bug where if an invalid `search[id]` was provided, then the custom order
would be ignored and the search would be returned with the default order
instead. Now if you use `search[order]=custom` without a valid
`search[id]` param, the search will return no results.
Fix the approver:, parent:, and pixiv: metatags not working correctly when negated:
* Fix -approver:<name> not including posts that don't have an approver (the approver_id is NULL)
* Fix -parent:<id> not including posts that don't have a parent (the parent_id is NULL)
* Fix -pixiv:<id> not including posts that aren't from Pixiv (the pixiv_id is NULL)
The problem lies how the equality operator is negated when the column contains NULL values;
`approver_id != 52664` doesn't match posts where the `approver_id` is NULL.
The search `approver:evazion` boils down to:
# Post.where(approver_id: 52664).to_sql
SELECT * FROM posts WHERE approver_id = 52664;
When that is negated with `-approver:evazion`, it becomes:
# Post.where(approver_id: 52664).invert_where.to_sql
SELECT * FROM posts WHERE approver_id != 52664;
But in SQL, `approver_id != 52664` doesn't match when the approver_id IS NULL, so the search doesn't
include posts without an approver.
We could use `a IS NOT DISTINCT FROM b` instead of `a = b`:
# Post.where(Post.arel_table[:approver_id].is_not_distinct_from(52664)).to_sql
SELECT * FROM posts WHERE approver_id IS NOT DISTINCT FROM 52664;
This way when it's inverted it becomes `IS DISTINCT FROM`:
# Post.where(Post.arel_table[:approver_id].is_not_distinct_from(52664)).invert_where.to_sql
SELECT * FROM posts WHERE approver_id IS NOT DISTINCT FROM 52664;
`approver_id IS DISTINCT FROM 52664` is like `approver_id != 52664`, except it matches when
approver_id is NULL [1].
This works correctly, however the problem is that `IS NOT DISTINCT FROM` can't use indexes because
of a long-standing Postgres limitation [2]. This makes searches too slow. So instead we do this:
# Post.where(approver_id: 52664).where.not(approver_id: nil).to_sql
SELECT * FROM posts WHERE approver_id = 52664 AND approver_id IS NOT NULL;
That way when negated it becomes:
# Post.where(approver_id: 52664).where.not(approver_id: nil).invert_where.to_sql
SELECT * FROM posts WHERE approver_id != 52664 OR approver_id IS NULL;
Which is the correct behavior.
[1] https://modern-sql.com/feature/is-distinct-from
[2] https://www.postgresql.org/message-id/6FC83909-5DB1-420F-9191-DBE533A3CEDE@excoventures.com
Show the following actions on the post events page:
* Post bans and unbans
* Post deletions and undeletions
* Thumbnail regenerations and IQDB regenerations
* Favorites moves
* Rating locks and unlocks
* Note locks and unlocks
Fixes#3825: Events/Moderation page for each post should show eventual ban actions
Fix IP unban actions and moderation report handled/rejected actions
being visible to non-moderators in the mod actions log.
Moderation report actions didn't leak the modreport itself, but it did
leak which moderator handled or rejected it.
Add a fix script to populate the mod_actions subject field by parsing
mod action descriptions. Most mod actions contain an ID, so finding the
subject is easy, but some don't. And some mod actions refer to deleted
objects, such as deleted posts or comments. In these cases the subject
will be null.
For IP bans, the mod action description only contains the IP, but it's
possible to have multiple bans for the same IP. So we look for IP bans
created by the same user, for the same IP, within the same time range.
For user bans, the mod action only contains the banned user's name and
the ban reason. This makes it difficult to find the banned user's ID in
some cases, because it's possible for the user to have changed their
name, and for the name change to have not been recorded, and for the
banner to have edited the ban reason, or for the ban to have been
deleted. So we try multiple things until we find the closest match.
Add a polymorphic `subject` field that records the subject of the mod
action. The subject is the post, user, comment, artist, etc the mod
action is for.
* The subject for the user ban and unban actions is the user, not the ban itself.
* The subject for the user feedback update and deletion actions is the user,
not the feedback itself.
* The subject for the post undeletion action is the post, not the approval itself.
* The subject for the move favorites action is the source post where the
favorites were moved from, not the destination post where the favorites
were moved to.
* The subject for the post permanent delete action is nil, because the
post itself is hard deleted.
* When a post is permanently deleted, all mod actions related to the
post are deleted as well.
* Add a global /post_events page that shows the history of all approvals,
disapprovals, flags, appeals, and replacements on a single page.
* Redesign the /posts/:id/events page to show all approval, disapproval,
flag, appeal, and replacement events for a single post (before it only
showed approvals, flags, and appeals).
* Remove the replacement history link from the post show page. Replacements
are now included in the post events page (closes#4948: Highlighed replacements).
* Add /post_approvals/:id and /post_replacements/:id routes (these are
used by the "Details" link on the post events page).
Make the following fields visible in API responses:
* ip_bans.ip_addr
* ip_geolocations.ip_addr
* ip_geolocations.network
* users.last_ip_addr (mod only)
* user_sessions.ip_addr
* api_keys.last_ip_address
* api_keys.permitted_ip_addresses
Before IP addresses were globally hidden in API responses because IPs were
present in a lot of tables and we didn't want to accidentally leak them.
Now that we've gotten rid of IPs from most tables, it's safe to unhide them.
Fix searching post disapprovals by user to use the new `visible_for_search`
mechanism. This fixes it so you can do subsearches on the user field, like this:
* https://danbooru.donmai.us/post_disapprovals?search[user][level]=32 (find disapprovals by Builder-level approvers)
Fix exception when viewing old post versions that contain null values in the
added_tags field: https://danbooru.donmai.us/post_versions?search[id]=1319677
There are 82 post versions with nulls in the added_tags fields and 26
with nulls in the removed_tags field.
Fix the search `flagger:evazion user:evazion` not returning the user's own self-flagged uploads.
Followup to a6e0872ce.
Fixes#4690: user profile 'flags' count links to /post_flags with different search criteria
* Remove the /comment/search page. Instead put the comment search form
on the same page as the search results, so you don't have to go back
and forth to update your search.
* Add an "Edited?" search option, for finding comments that have been edited.
* Add options for sorting by oldest comments and lowest scoring comments.
* Let the user see their own username when viewing their own deleted comments.
* Don't hide the creator_id field from the comment creator in the API.
* Hide the score, do_not_bump_post, and is_sticky fields for deleted
comments in the HTML and in the API, unless the user is a moderator.
* Hide the "..." popup menu on deleted comments, unless the user is a moderator.
This is so that when a user looks at their own comment history, their
name isn't hidden from them on their own deleted comments. This may confuse
users however into thinking their name is still visible to other users.
* Fix it so non-moderators can't search deleted comments using the
`updater`, `body`, `score`, `do_not_bump_post`, or `is_sticky` fields.
Searching for these fields will exclude deleted comments.
* Fix it so non-moderators can search for their own deleted comments using the
`creator` field, but not for deleted comments belonging to other users.
* Fix it so that if a regular user searches `commenter:<username>`, they
can only see posts with undeleted comments by that user. If a moderator or
the commenter themselves searches `commenter:<username>`, they can see all
posts the user has commented on, including posts with deleted comments.
* Fix it so the comment count on user profiles only counts visible
comments. Regular users can only see the number of undeleted comments
a user has, while moderators and the commenter themselves can see the
total number of comments.
Known issue:
* It's still possible to order deleted comments by score, which can let
you infer the score of deleted comments.
Switch `/artist_commentaries?search[text_matches]` to do a full-text
search instead of a substring search. This is more consistent with text
search in other places.
The previous commit changed it so that `/pools?search[name_matches]`
does a full-text search. So for example, `search[name_matches]=smiling`
will now match pool names containing any of the words "smiling",
"smile", "smiles", or "smiled".
This commit adds a `/pools?search[name_contains]` param that does what
`name_matches` did before, and switches to it in search forms. So for
example, `search[name_contains]=smiling` will only match pool names
containing the exact substring "smiling".
This change is so that `<field>_matches` works consistently across the
site, and so that it's possible to search pool names by either an exact
substring match, or by a looser natural language match.
This is a minor breaking API change. API users can replace
`/pools?search[name_matches]` with `/pools?search[name_contains]` to get
the same behavior as before. The same applies to /favorite_groups.
Standardize it so that all fields of type `text` are searchable with
`search[<field>_matches]`.
Before, the `<field>_matches` param was handled manually and some fields
were left out or handled inconsistently. Now it applies to all columns
of type `text`.
This does a full-text search on the field, so for example, searching
`/artist_commentaries?search[translated_description_matches]=smiling`
will match translated commentaries containing either the word "smiling",
"smiles", "smiled", or "smile".
Note that this only applies to columns defined as type `text`, not to
columns defined as `character varying`. The difference is that `text` is
used for fields containing free-form natural language, such as comments,
notes, forum posts, wiki pages, pool descriptions, etc, while `character
varying` is used for short strings not containing free-form language,
such as tag names, wiki page titles, urls, status fields, etc.
API changes:
* Add the `search[original_title_matches]`, `search[original_description_matches]`,
`search[translated_title_matches]`, `search[translated_description_matches]` params
to /artist_commentaries and /artist_commentary_versions.
* Remove the `search[name_matches]` and `search[group_name_matches]` params from /artist_versions.
* Remove the `search[title_matches]` param from /wiki_page_versions.
* Change the `search[name_matches]` param on /pools, /favorite_groups, and /pool_versions
to do a full-text search instead of a substring match.
Fix a bug where, when a mod searched for their own flags, they couldn't
see their own self-flagged uploads.
Fix a bug where, when a mod viewed their own flags, they couldn't see
the flagger name for their own self-flagged uploads.
This also makes it so you can do things like `/post_flags?search[creator][level]=20`
to search flags by user level.
Add a `visible_for_search` method to ApplicationPolicy that lets us
define which fields a user is allowed to search for.
For example, when a normal user searches for post flags by flagger name,
they're only allowed to see their own flags, not flags by other users.
But when a mod searches for flags by flagger name, they're allowed to
see all flags, except for flags on their own uploads.
This framework lets us define these rules in the `visible_for_search`
method in the model's policy class, rather than as special cases in the
`search` method of each model.
Move the `search_attributes` helper methods inside the Searchable
concern into a SearchContext helper class. This is so we don't have to
pass `params` and `current_user` down through a bunch of different
helper methods, and so that these private helper methods don't pollute
the object's namespace.
Fix various columns to be either `character varying` or `text`,
depending on what kind of text is stored in the column. `text` is used
for columns that contain free-form natural language, like pool and forum
topic titles, while `character varying` is used for short strings that
don't contain free-form text, like URLs and status fields.
Both types are treated the same by Postgres; the only difference is how
we treat them in Rails. In edit forms, `text` fields use multi-line
textboxes, while `character varying` fields use single-line inputs. And
during search, we allow `text` fields to be searched using full-text
search, but not `character varying` fields.
Don't allow GET requests to pass the request params in the body instead
of in the URL. While Rails can handle GET params passed in the body, it
goes against spec and it may cause problems if the response is a redirect
and the client doesn't send the body params when following the redirect.
This may be a breaking change for broken API clients who were sending
GET params in the body instead of in the URL. This can happen when people
use HTTP libraries incorrectly.
Add a fix script to delete all accounts with invalid usernames. Also
change it so the owner-level user can delete accounts belonging to other
users.
Users who have logged in in the last year and who have a valid email
address will be given a one week warning. After that all accounts with
invalid names will be deleted. Anyone who has visited the site in the
last 6 months will have already seen a warning page that their name must
be changed to keep using the site.
Fix mod actions to use the same message format everywhere.
Before mod actions were formatted in various inconsistent ways:
* "deleted post #1234"
* "comment #1234 updated by <user>"
* "<user> updated forum #1234"
* "<user> level changed Member -> Builder"
Now all mod actions consistently use this format:
* "deleted post #1234"
* "updated comment #1234"
* "updated forum #1234"
* "promoted <user> from Member to Builder"
This way mod actions are formatted consistently with other actions on
the /user_actions page, where everything is written as "<user> did X".
Also add a fix script to fix existing mod actions.
Allow approvers to bypass the rule that you can't flag a post again if
it was flagged less than 3 days ago. This rule was intended to prevent
flag warring among regular users, which hopefully shouldn't be a problem
among approvers. It was also useless because approvers could always
just directly delete the post even if they couldn't flag it.
Allowing approvers to reflag posts allows them to reinstate flags that
were accidentally approved.
Remove the `CurrentUser.ip_addr` global variable and replace it with
`request.remote_ip`. Before we had to track the current user's IP in a
global variable so that when we edited a post for example, we could pass
down the user's IP to the model and save it in the post_versions table.
Now that we now longer save IPs in version tables, we don't need a global
variable to get access to the current user's IP outside of controllers.
Set various IP address columns to nullable in preparation for dropping them.
In production, some of these tables already contained null values even
though it violated the constraint.
Remove the /ip_addresses page. This page allowed moderators to search
users by IP, and to see recent activity tied to an IP. However, it was
limited to IPs tied to uploads, comments, dmails, artist edits, note
edits, and wiki edits.
Remove this page because it was limited in scope and because there are
better ways of doing what it did. The /user_events page is better at
catching sockpuppets because it tracks IPs for every login, not just for
certain types of edits. And the /user_actions page is better at
monitoring user activity because it shows all activity associated with
an account, not just for certain types of edits.
Removing this allows us to drop IP addresses from all tables besides the
user_events table. This is good because these IPs are no longer necessary
for any purpose, and because storing them forever is a liability.
evazion
evazion