Fix a bug where searching for a negated tag inside the modqueue would show
active posts.
The bug was that in a search like this:
Post.in_modqueue.user_tag_match("-solo")
The `in_modqueue` condition would get sucked inside the tag search and negated
when we tried to apply the negation operator to the "solo" tag. So effectively
the `in_modqueue` condition would get negated and we would end up searching for
everything not in the modqueue.
* Optimize status:modqueue and status:unmoderated searches. This brings them down from
taking 500ms-1000ms per search to ~5ms.
* Change status:unmoderated so that it only filters out the user's disapproved posts, not
the user's own uploads or past approvals. Now it's equivalent to `status:modqueue -disapproved:evazion`,
whereas before it was equivalent to `status:modqueue -disapproved:evazion -approver:evazion -user:evazion`.
Filtering out the user's own uploads and approvals was slow and usually unnecessary,
since for most users it's rare for their own uploads or approvals to reenter the modqueue.
Before status:modqueue did this:
SELECT * FROM posts WHERE is_pending = TRUE OR is_flagged = TRUE OR (is_deleted = TRUE AND id IN (SELECT post_id FROM post_appeals WHERE status = 0))
Now we do this:
SELECT * FROM posts WHERE id IN (SELECT id FROM posts WHERE is_pending = TRUE UNION ALL SELECT id FROM posts WHERE is_flagged = TRUE UNION ALL SELECT id FROM posts WHERE id IN (SELECT post_id FROM post_appeals WHERE status = 0))
Postgres had a bad time with the "pending or flagged or has a pending appeal" clause because
it didn't know that posts can only be in one state at a time, so it overestimated how many
posts would be returned and chose a seq scan. Replacing the OR with a UNION avoids this.
Fix the approver:, parent:, and pixiv: metatags not working correctly when negated:
* Fix -approver:<name> not including posts that don't have an approver (the approver_id is NULL)
* Fix -parent:<id> not including posts that don't have a parent (the parent_id is NULL)
* Fix -pixiv:<id> not including posts that aren't from Pixiv (the pixiv_id is NULL)
The problem lies how the equality operator is negated when the column contains NULL values;
`approver_id != 52664` doesn't match posts where the `approver_id` is NULL.
The search `approver:evazion` boils down to:
# Post.where(approver_id: 52664).to_sql
SELECT * FROM posts WHERE approver_id = 52664;
When that is negated with `-approver:evazion`, it becomes:
# Post.where(approver_id: 52664).invert_where.to_sql
SELECT * FROM posts WHERE approver_id != 52664;
But in SQL, `approver_id != 52664` doesn't match when the approver_id IS NULL, so the search doesn't
include posts without an approver.
We could use `a IS NOT DISTINCT FROM b` instead of `a = b`:
# Post.where(Post.arel_table[:approver_id].is_not_distinct_from(52664)).to_sql
SELECT * FROM posts WHERE approver_id IS NOT DISTINCT FROM 52664;
This way when it's inverted it becomes `IS DISTINCT FROM`:
# Post.where(Post.arel_table[:approver_id].is_not_distinct_from(52664)).invert_where.to_sql
SELECT * FROM posts WHERE approver_id IS NOT DISTINCT FROM 52664;
`approver_id IS DISTINCT FROM 52664` is like `approver_id != 52664`, except it matches when
approver_id is NULL [1].
This works correctly, however the problem is that `IS NOT DISTINCT FROM` can't use indexes because
of a long-standing Postgres limitation [2]. This makes searches too slow. So instead we do this:
# Post.where(approver_id: 52664).where.not(approver_id: nil).to_sql
SELECT * FROM posts WHERE approver_id = 52664 AND approver_id IS NOT NULL;
That way when negated it becomes:
# Post.where(approver_id: 52664).where.not(approver_id: nil).invert_where.to_sql
SELECT * FROM posts WHERE approver_id != 52664 OR approver_id IS NULL;
Which is the correct behavior.
[1] https://modern-sql.com/feature/is-distinct-from
[2] https://www.postgresql.org/message-id/6FC83909-5DB1-420F-9191-DBE533A3CEDE@excoventures.com
Show the following actions on the post events page:
* Post bans and unbans
* Post deletions and undeletions
* Thumbnail regenerations and IQDB regenerations
* Favorites moves
* Rating locks and unlocks
* Note locks and unlocks
Fixes#3825: Events/Moderation page for each post should show eventual ban actions
Fix IP unban actions and moderation report handled/rejected actions
being visible to non-moderators in the mod actions log.
Moderation report actions didn't leak the modreport itself, but it did
leak which moderator handled or rejected it.
Add a polymorphic `subject` field that records the subject of the mod
action. The subject is the post, user, comment, artist, etc the mod
action is for.
* The subject for the user ban and unban actions is the user, not the ban itself.
* The subject for the user feedback update and deletion actions is the user,
not the feedback itself.
* The subject for the post undeletion action is the post, not the approval itself.
* The subject for the move favorites action is the source post where the
favorites were moved from, not the destination post where the favorites
were moved to.
* The subject for the post permanent delete action is nil, because the
post itself is hard deleted.
* When a post is permanently deleted, all mod actions related to the
post are deleted as well.
* Add a global /post_events page that shows the history of all approvals,
disapprovals, flags, appeals, and replacements on a single page.
* Redesign the /posts/:id/events page to show all approval, disapproval,
flag, appeal, and replacement events for a single post (before it only
showed approvals, flags, and appeals).
* Remove the replacement history link from the post show page. Replacements
are now included in the post events page (closes#4948: Highlighed replacements).
* Add /post_approvals/:id and /post_replacements/:id routes (these are
used by the "Details" link on the post events page).
Make the following fields visible in API responses:
* ip_bans.ip_addr
* ip_geolocations.ip_addr
* ip_geolocations.network
* users.last_ip_addr (mod only)
* user_sessions.ip_addr
* api_keys.last_ip_address
* api_keys.permitted_ip_addresses
Before IP addresses were globally hidden in API responses because IPs were
present in a lot of tables and we didn't want to accidentally leak them.
Now that we've gotten rid of IPs from most tables, it's safe to unhide them.
Fix searching post disapprovals by user to use the new `visible_for_search`
mechanism. This fixes it so you can do subsearches on the user field, like this:
* https://danbooru.donmai.us/post_disapprovals?search[user][level]=32 (find disapprovals by Builder-level approvers)
Fix the search `flagger:evazion user:evazion` not returning the user's own self-flagged uploads.
Followup to a6e0872ce.
Fixes#4690: user profile 'flags' count links to /post_flags with different search criteria
* Remove the /comment/search page. Instead put the comment search form
on the same page as the search results, so you don't have to go back
and forth to update your search.
* Add an "Edited?" search option, for finding comments that have been edited.
* Add options for sorting by oldest comments and lowest scoring comments.
* Fix it so non-moderators can't search deleted comments using the
`updater`, `body`, `score`, `do_not_bump_post`, or `is_sticky` fields.
Searching for these fields will exclude deleted comments.
* Fix it so non-moderators can search for their own deleted comments using the
`creator` field, but not for deleted comments belonging to other users.
* Fix it so that if a regular user searches `commenter:<username>`, they
can only see posts with undeleted comments by that user. If a moderator or
the commenter themselves searches `commenter:<username>`, they can see all
posts the user has commented on, including posts with deleted comments.
* Fix it so the comment count on user profiles only counts visible
comments. Regular users can only see the number of undeleted comments
a user has, while moderators and the commenter themselves can see the
total number of comments.
Known issue:
* It's still possible to order deleted comments by score, which can let
you infer the score of deleted comments.
Switch `/artist_commentaries?search[text_matches]` to do a full-text
search instead of a substring search. This is more consistent with text
search in other places.
The previous commit changed it so that `/pools?search[name_matches]`
does a full-text search. So for example, `search[name_matches]=smiling`
will now match pool names containing any of the words "smiling",
"smile", "smiles", or "smiled".
This commit adds a `/pools?search[name_contains]` param that does what
`name_matches` did before, and switches to it in search forms. So for
example, `search[name_contains]=smiling` will only match pool names
containing the exact substring "smiling".
This change is so that `<field>_matches` works consistently across the
site, and so that it's possible to search pool names by either an exact
substring match, or by a looser natural language match.
This is a minor breaking API change. API users can replace
`/pools?search[name_matches]` with `/pools?search[name_contains]` to get
the same behavior as before. The same applies to /favorite_groups.
Standardize it so that all fields of type `text` are searchable with
`search[<field>_matches]`.
Before, the `<field>_matches` param was handled manually and some fields
were left out or handled inconsistently. Now it applies to all columns
of type `text`.
This does a full-text search on the field, so for example, searching
`/artist_commentaries?search[translated_description_matches]=smiling`
will match translated commentaries containing either the word "smiling",
"smiles", "smiled", or "smile".
Note that this only applies to columns defined as type `text`, not to
columns defined as `character varying`. The difference is that `text` is
used for fields containing free-form natural language, such as comments,
notes, forum posts, wiki pages, pool descriptions, etc, while `character
varying` is used for short strings not containing free-form language,
such as tag names, wiki page titles, urls, status fields, etc.
API changes:
* Add the `search[original_title_matches]`, `search[original_description_matches]`,
`search[translated_title_matches]`, `search[translated_description_matches]` params
to /artist_commentaries and /artist_commentary_versions.
* Remove the `search[name_matches]` and `search[group_name_matches]` params from /artist_versions.
* Remove the `search[title_matches]` param from /wiki_page_versions.
* Change the `search[name_matches]` param on /pools, /favorite_groups, and /pool_versions
to do a full-text search instead of a substring match.
Fix a bug where, when a mod searched for their own flags, they couldn't
see their own self-flagged uploads.
Fix a bug where, when a mod viewed their own flags, they couldn't see
the flagger name for their own self-flagged uploads.
This also makes it so you can do things like `/post_flags?search[creator][level]=20`
to search flags by user level.
Fix mod actions to use the same message format everywhere.
Before mod actions were formatted in various inconsistent ways:
* "deleted post #1234"
* "comment #1234 updated by <user>"
* "<user> updated forum #1234"
* "<user> level changed Member -> Builder"
Now all mod actions consistently use this format:
* "deleted post #1234"
* "updated comment #1234"
* "updated forum #1234"
* "promoted <user> from Member to Builder"
This way mod actions are formatted consistently with other actions on
the /user_actions page, where everything is written as "<user> did X".
Also add a fix script to fix existing mod actions.
Allow approvers to bypass the rule that you can't flag a post again if
it was flagged less than 3 days ago. This rule was intended to prevent
flag warring among regular users, which hopefully shouldn't be a problem
among approvers. It was also useless because approvers could always
just directly delete the post even if they couldn't flag it.
Allowing approvers to reflag posts allows them to reinstate flags that
were accidentally approved.
Remove the `CurrentUser.ip_addr` global variable and replace it with
`request.remote_ip`. Before we had to track the current user's IP in a
global variable so that when we edited a post for example, we could pass
down the user's IP to the model and save it in the post_versions table.
Now that we now longer save IPs in version tables, we don't need a global
variable to get access to the current user's IP outside of controllers.
Make all forum post votes visible to everyone.
When forum votes were first introduced, it was technically possible to
vote on any forum post, including on posts in mod-only threads.
Accordingly, forum post votes were only visible if the forum post itself
was visible. However, there doesn't actually exist any votes on private
forum posts, and trying to filter them out makes the /user_actions page
much slower, so just make them visible to everyone.
Allow moderators to see all events on the /user_events page. Before only
admins could see when a user changed their email, changed their
password, or had a failed login attempt. Now moderators can see these
events too.
Filtering these events out made the /user_actions page slower, and it
wasn't really necessary since merely knowing that a user changed their
email or password isn't that much more sensitive than knowing when they
logged in or out.
Add a /user_actions page. This page shows you a global timeline of
(almost) all activity on the site, including uploads, comments, votes,
edits, forum posts, and so on.
The main things it doesn't include are post edits, pool edits, and
favorites (posts and pools live in a separate database, and favorites
don't have the timestamps we need for ordering).
This page is useful for moderation purposes because it lets you see a
history of almost all of a user's activity on a single page.
Currently this page is mod-only. In the future it will be open to all
users, so you can view the history of your own site activity, or the
activity of others.
Unlike other models, notes had a special rule where if you deleted or
undeleted a note, it would always create a new version instead of
merging it into the previous version. Remove this rule since it didn't
have a purpose and it was inconsistent with other versioned models.
Track the history of the tag `category` and `is_deprecated` fields in
the `tag_versions` table.
Adds generic Versionable and VersionFor concerns that encapsulate most
of the history tracking logic. These concerns are designed to make it
easy to add history to any model.
There are a couple notable differences between tag versions and other versions:
* There is no 1 hour edit merge window. All changes to the `category`
and `is_deprecated` fields produce a new version in the tag history.
* New versions aren't created when a tag is created. Versions are only
created when a tag is edited for the first time. The tag's initial
version isn't created until *after* the tag is edited for the first time.
For example, if you change the category of a tag that was last updated
10 years ago, that will create an initial version of the tag backdated
to 10 years ago, plus a new version for your edit.
This is for a few reasons:
* So that we don't have to create new tag versions every time a new tag
is created. This would be wasteful because most tags never have their
category or deprecation status change.
* So that if you make a typo tag, your name isn't recorded in the tag's
history forever.
* So that we can create new tags in various places without having to know
who created the tag (which may be unknown if the current user isn't set).
* Because we don't know the full history of most tags, so we have to
deal with incomplete histories anyway.
This has a few important consequences:
* Most tags won't have any tag versions. They only gain tag versions if
they're edited.
* You can't track /tag_versions to see newly created tags. It only
shows changes to already existing tags.
* Tag version IDs won't be in strict chronological order. Higher IDs may
have created_at timestamps before lower IDs. For example, if you
change the category of a tag that is 10 years old, that will create an
initial version with a high ID, but with a created_at timestamp dated
to 10 years ago.
Fixes#4402: Track tag category changes
Fixup for 015c6dc7d. Show a warning about failure to add a tag instead
of raising an exception when trying to tag a post with `char:copy:foo`.
This tries to create a tag named `copy:foo` then set the category to
character, which doesn't work because `copy:foo` isn't a valid tag name.
Drop the ability to write e.g. `create alias foo -> char:bar` in a BUR
to change the tag's type as a side effect. You can only use these
tag type prefixes in tag edits now.
This feature was only intended to be used in tag edits. The fact it
worked elsewhere was unintended behavior.
This feature was problematic because it relied on `Tag.find_or_create_by_name`
automagically changing the tag's category when the tag name contained a
tag category prefix, e.g. `char:hatsune_miku`. This meant that merely
looking up a tag could have the side effect of changing its category.
It was also bad because `find_or_create_by_name` had a hidden dependency
on the current user, which may not be set or available in all contexts.
Fix a bug where the word `lazy` wasn't highlighted in `don't_say_"lazy"`
when searching for `lazy`. The bug was that punctuation surrounding
words wasn't properly split from the word.
Switch autocomplete to match individual words in the tag, instead of
only matching the start of the tag.
For example, "hair" matches any tag containing the word "hair", not just tags
starting with "hair". "long_hair" matches all tags containing the words "long"
and "hair", which includes "very_long_hair" and "absurdly_long_hair".
Words can be in any order and words can be left out. So "closed_eye" matches
"one_eye_closed". "asuka_langley_souryuu" matches "souryuu_asuka_langley".
This has several advantages:
* You can search characters by first name. For example, "miku" matches "hatsune_miku".
"zelda" matches both "princess_zelda" and "the_legend_of_zelda".
* You can find the right tag even if you get the word order wrong, or forget a word.
For example, "eyes_closed" matches "closed_eyes". "hair_over_eye" matches "hair_over_one_eye".
* You can find more related tags. For example, searching "skirt" shows all tags
containing the word "skirt", not just tags starting with "skirt".
The downside is this may break muscle memory by changing the autocomplete order of
some tags. This is an acceptable trade-off.
You can get the old behavior by writing a "*" at the end of the tag. For
example, searching "skirt*" gives the same results as before.
