Add ability to upload .webp images.
Animated WebP images aren't supported. This is because they aren't
supported by FFmpeg yet[1], so generating thumbnails and samples for
them would be more complicated than for other formats.
[1]: https://trac.ffmpeg.org/ticket/4907
Features of AVIF include:
* Lossless and lossy compression.
* High dynamic range (HDR) images
* Wide color gamut images (i.e. 10- and 12-bit color depths)
* Transparency (through alpha planes).
* Animations (with an optional cover image).
* Auxiliary image sequences, where the file contains a single primary
image and a short secondary video, like Apple's Live Photos.
* Metadata rotation, mirroring, and cropping.
The AVIF format is still relatively new and some of these features aren't well
supported by browsers or other software:
* Animated AVIFs aren't supported by Firefox or by libvips.
* HDR images aren't supported by Firefox.
* Rotated, mirrored, and cropped AVIFs aren't supported by Firefox or Chrome.
* Image grids, where the file contains multiple images that are tiled
together into one big image, aren't supported by Firefox.
* AVIF as a whole has only been supported for a year or two by Chrome
and Firefox, and less than a year by Safari.
For these reasons, only basic AVIFs that don't use animation, rotation,
cropping, or image grids can be uploaded.
Update email validation rules to disallow the percent character (e.g.
`foo%bar@gmail.com`) and names ending with a period (e.g. `foo.@gmail.com`).
Names ending with a period are invalid according to the RFCs and cause
`Mail::Address.new` to raise an exception.
The percent character is technically legal, but only one email used it
and it was probably a typo.
* Make it an error to supply empty API credentials, like this:
`https://danbooru.donmai.us/posts.json?login=&api_key=`. Some clients
did this for some reason.
* Make it so that the `login` and `api_key` params are only allowed as
URL params, not as POST or PUT body params. Allowing them as body
params could interfere with the `PUT /api_keys/:id` endpoint, which
takes an `api_key` param.
Try to automatically fix various kind of typos and common mistakes in
email addresses when a user creates a new account. It's common for users
to signup with addresses like `name@gmai.com`, which leads to bounces
when we try to send the welcome email.
Add support for uploading posts from Gelbooru. Note that the translated
tags will include both the Gelbooru tags and the tags from the Gelbooru
post's source. The commentary and artist information will also be taken
from the Gelbooru post's source. The source of the Danbooru post however
will be left as the Gelbooru post itself, not as the Gelbooru post's source.
Remove the last remaining uses of the PixivUgoiraFrameData model. As of
32bfb8407, Ugoira frame data is now stored in the MediaMetadata model,
under the `Ugoira:FrameDelays` EXIF field.
The pixiv_ugoira_frame_data table still exists, but it can be removed
after this commit is deployed.
Fixes#5264: Error when replacing with ugoira.
* Fixed a bug where manga posts with a single tag would raise an error
* Fixed a bug where dic.nicovideo.jp/oekaki posts weren't uploadable due
to SSL issues
* Added support for more manga corner cases
Fix not being able to use the full set of search operators on polymorphic `model_id` and
`model_type` attributes. Before things like `search[model_type]=Post` worked, but
`search[model_type_not_eq]=Post` or other `model_type_*` operators didn't.
Fix a bug where searching for a negated tag inside the modqueue would show
active posts.
The bug was that in a search like this:
Post.in_modqueue.user_tag_match("-solo")
The `in_modqueue` condition would get sucked inside the tag search and negated
when we tried to apply the negation operator to the "solo" tag. So effectively
the `in_modqueue` condition would get negated and we would end up searching for
everything not in the modqueue.
* Optimize status:modqueue and status:unmoderated searches. This brings them down from
taking 500ms-1000ms per search to ~5ms.
* Change status:unmoderated so that it only filters out the user's disapproved posts, not
the user's own uploads or past approvals. Now it's equivalent to `status:modqueue -disapproved:evazion`,
whereas before it was equivalent to `status:modqueue -disapproved:evazion -approver:evazion -user:evazion`.
Filtering out the user's own uploads and approvals was slow and usually unnecessary,
since for most users it's rare for their own uploads or approvals to reenter the modqueue.
Before status:modqueue did this:
SELECT * FROM posts WHERE is_pending = TRUE OR is_flagged = TRUE OR (is_deleted = TRUE AND id IN (SELECT post_id FROM post_appeals WHERE status = 0))
Now we do this:
SELECT * FROM posts WHERE id IN (SELECT id FROM posts WHERE is_pending = TRUE UNION ALL SELECT id FROM posts WHERE is_flagged = TRUE UNION ALL SELECT id FROM posts WHERE id IN (SELECT post_id FROM post_appeals WHERE status = 0))
Postgres had a bad time with the "pending or flagged or has a pending appeal" clause because
it didn't know that posts can only be in one state at a time, so it overestimated how many
posts would be returned and chose a seq scan. Replacing the OR with a UNION avoids this.
Support searches like the following:
* score:<0,>100 (equivalent to `score:<0 or score:>100`)
* score:5,10..15,>20 (equivalent to `score:5 or score:10..15 or score:>20`)
* id:5,10..15,20..25,30 (equivalent to `id:5 or id:10..15 or id:20..25 or id:30`)
This also works inside the `search[id]` URL parameter, and inside other numeric
URL search parameters.
Fix the approver:, parent:, and pixiv: metatags not working correctly when negated:
* Fix -approver:<name> not including posts that don't have an approver (the approver_id is NULL)
* Fix -parent:<id> not including posts that don't have a parent (the parent_id is NULL)
* Fix -pixiv:<id> not including posts that aren't from Pixiv (the pixiv_id is NULL)
The problem lies how the equality operator is negated when the column contains NULL values;
`approver_id != 52664` doesn't match posts where the `approver_id` is NULL.
The search `approver:evazion` boils down to:
# Post.where(approver_id: 52664).to_sql
SELECT * FROM posts WHERE approver_id = 52664;
When that is negated with `-approver:evazion`, it becomes:
# Post.where(approver_id: 52664).invert_where.to_sql
SELECT * FROM posts WHERE approver_id != 52664;
But in SQL, `approver_id != 52664` doesn't match when the approver_id IS NULL, so the search doesn't
include posts without an approver.
We could use `a IS NOT DISTINCT FROM b` instead of `a = b`:
# Post.where(Post.arel_table[:approver_id].is_not_distinct_from(52664)).to_sql
SELECT * FROM posts WHERE approver_id IS NOT DISTINCT FROM 52664;
This way when it's inverted it becomes `IS DISTINCT FROM`:
# Post.where(Post.arel_table[:approver_id].is_not_distinct_from(52664)).invert_where.to_sql
SELECT * FROM posts WHERE approver_id IS NOT DISTINCT FROM 52664;
`approver_id IS DISTINCT FROM 52664` is like `approver_id != 52664`, except it matches when
approver_id is NULL [1].
This works correctly, however the problem is that `IS NOT DISTINCT FROM` can't use indexes because
of a long-standing Postgres limitation [2]. This makes searches too slow. So instead we do this:
# Post.where(approver_id: 52664).where.not(approver_id: nil).to_sql
SELECT * FROM posts WHERE approver_id = 52664 AND approver_id IS NOT NULL;
That way when negated it becomes:
# Post.where(approver_id: 52664).where.not(approver_id: nil).invert_where.to_sql
SELECT * FROM posts WHERE approver_id != 52664 OR approver_id IS NULL;
Which is the correct behavior.
[1] https://modern-sql.com/feature/is-distinct-from
[2] https://www.postgresql.org/message-id/6FC83909-5DB1-420F-9191-DBE533A3CEDE@excoventures.com
Add a polymorphic `subject` field that records the subject of the mod
action. The subject is the post, user, comment, artist, etc the mod
action is for.
* The subject for the user ban and unban actions is the user, not the ban itself.
* The subject for the user feedback update and deletion actions is the user,
not the feedback itself.
* The subject for the post undeletion action is the post, not the approval itself.
* The subject for the move favorites action is the source post where the
favorites were moved from, not the destination post where the favorites
were moved to.
* The subject for the post permanent delete action is nil, because the
post itself is hard deleted.
* When a post is permanently deleted, all mod actions related to the
post are deleted as well.
* Add a global /post_events page that shows the history of all approvals,
disapprovals, flags, appeals, and replacements on a single page.
* Redesign the /posts/:id/events page to show all approval, disapproval,
flag, appeal, and replacement events for a single post (before it only
showed approvals, flags, and appeals).
* Remove the replacement history link from the post show page. Replacements
are now included in the post events page (closes#4948: Highlighed replacements).
* Add /post_approvals/:id and /post_replacements/:id routes (these are
used by the "Details" link on the post events page).
Make the following fields visible in API responses:
* ip_bans.ip_addr
* ip_geolocations.ip_addr
* ip_geolocations.network
* users.last_ip_addr (mod only)
* user_sessions.ip_addr
* api_keys.last_ip_address
* api_keys.permitted_ip_addresses
Before IP addresses were globally hidden in API responses because IPs were
present in a lot of tables and we didn't want to accidentally leak them.
Now that we've gotten rid of IPs from most tables, it's safe to unhide them.
Fix the search `flagger:evazion user:evazion` not returning the user's own self-flagged uploads.
Followup to a6e0872ce.
Fixes#4690: user profile 'flags' count links to /post_flags with different search criteria
* Fix it so non-moderators can't search deleted comments using the
`updater`, `body`, `score`, `do_not_bump_post`, or `is_sticky` fields.
Searching for these fields will exclude deleted comments.
* Fix it so non-moderators can search for their own deleted comments using the
`creator` field, but not for deleted comments belonging to other users.
* Fix it so that if a regular user searches `commenter:<username>`, they
can only see posts with undeleted comments by that user. If a moderator or
the commenter themselves searches `commenter:<username>`, they can see all
posts the user has commented on, including posts with deleted comments.
* Fix it so the comment count on user profiles only counts visible
comments. Regular users can only see the number of undeleted comments
a user has, while moderators and the commenter themselves can see the
total number of comments.
Known issue:
* It's still possible to order deleted comments by score, which can let
you infer the score of deleted comments.
The previous commit changed it so that `/pools?search[name_matches]`
does a full-text search. So for example, `search[name_matches]=smiling`
will now match pool names containing any of the words "smiling",
"smile", "smiles", or "smiled".
This commit adds a `/pools?search[name_contains]` param that does what
`name_matches` did before, and switches to it in search forms. So for
example, `search[name_contains]=smiling` will only match pool names
containing the exact substring "smiling".
This change is so that `<field>_matches` works consistently across the
site, and so that it's possible to search pool names by either an exact
substring match, or by a looser natural language match.
This is a minor breaking API change. API users can replace
`/pools?search[name_matches]` with `/pools?search[name_contains]` to get
the same behavior as before. The same applies to /favorite_groups.
Move the `search_attributes` helper methods inside the Searchable
concern into a SearchContext helper class. This is so we don't have to
pass `params` and `current_user` down through a bunch of different
helper methods, and so that these private helper methods don't pollute
the object's namespace.
