Commit Graph
100 Commits
Author SHA1 Message Date
evazion b94cb7d824 emails: include logging information in email headers.
Log the following information in email headers:

* X-Danbooru-User: the user's name and ID.
* X-Danbooru-IP: the user's IP.
* X-Danbooru-Session: the users' session ID.
* X-Danbooru-URL: the page that triggered the email.
* X-Danbooru-Job-Id: the ID of the background job that sent the email.
* X-Danbooru-Enqueued-At: when the email was queued as a background job.
* X-Danbooru-Dmail: for Dmail notifications, the link to the Dmail.
* X-Request-Id: the request ID of the HTTP request that triggered the email.

Also make it so we log an event in the APM when we send an email.
2022-09-29 04:36:11 -05:00
evazion ed9986def6 emails: fix one-click unsubscription.
Fix it so that emails are (hopefully) able to show the one-click
unsubscribe button in Gmail and other mail providers that support the
List-Unsubscribe header. This way users can unsubscribe instead of
marking emails as spam.

* Add the List-Unsubscribe-Post header.
* Fix the disable email notifications endpoint to support POST as well as DELETE requests.
* Fix the disable email notifications endpoint to disable XSRF protection (we don't need
  users to be logged in because we use a signed URL instead).

https://www.rfc-editor.org/rfc/rfc8058#section-3.1
https://www.rfc-editor.org/rfc/rfc8058#section-8.1
2022-09-29 04:36:11 -05:00
evazion 266fed1b99 modqueue: fix incorrect page counts when filtering by tag.
Fix a bug where the modqueue didn't show the correct page count when
filtering by tag. Before we showed the total number of posts in the
modqueue, instead of the actual number of posts for your search. This
was for speed reasons, but with recent fixes it should be less of a
problem to count the actual number of pages for the search.
2022-09-28 00:32:29 -05:00
evazion 4c03ea5be3 Fix #5132: Modqueue displays active posts when excluding any search term
Fix a bug where searching for a negated tag inside the modqueue would show
active posts.

The bug was that in a search like this:

    Post.in_modqueue.user_tag_match("-solo")

The `in_modqueue` condition would get sucked inside the tag search and negated
when we tried to apply the negation operator to the "solo" tag. So effectively
the `in_modqueue` condition would get negated and we would end up searching for
everything not in the modqueue.
2022-09-28 00:29:50 -05:00
evazion 59f166a637 Fix #5057: Modqueue: filtering by tag breaks ordering
Fix the order dropdown box on the modqueue page not working when filtering by tag.

This happened because when you do a tag search, the default order is set to `ORDER BY posts.id DESC`.
When you applied another order with the dropdown box, the new order would be tacked on to the old
ordering as a tiebreaker instead of replacing it, producing e.g. `ORDER BY posts.id DESC, queued_at DESC`
instead of `ORDER BY queued_at DESC`. The default order would always win because `posts.id` is
unique and doesn't have ties.

The fix is to have orders always override the previous order instead of adding to it.

Note that now if you use an `order:`, `ordfav:`, or `ordpool:` metatag in the search box on the
modqueue page, they will always be ignored in favor of the dropdown box.
2022-09-28 00:29:50 -05:00
evazion 91c8038685 Fix #4606: Add only= support for modqueue. 2022-09-28 00:29:50 -05:00
evazion 04b004d2a4 Fix #4586: Wrong paginator for status:unmoderated
Don't cache the page count for status:unmoderated, status:modqueue, status:pending, status:flagged,
or status:appealed searches. Before the page count was cached for 5 minutes, which could quickly
become out of date as a user went through the modqueue. After bd21b4a86, they're now fast enough
that we no longer need to cache them.
2022-09-28 00:29:50 -05:00
evazion f49b3c439f posts: optimize modqueue page, status:modqueue, and status:unmoderated searches.
* Optimize status:modqueue and status:unmoderated searches. This brings them down from
  taking 500ms-1000ms per search to ~5ms.

* Change status:unmoderated so that it only filters out the user's disapproved posts, not
  the user's own uploads or past approvals. Now it's equivalent to `status:modqueue -disapproved:evazion`,
  whereas before it was equivalent to `status:modqueue -disapproved:evazion -approver:evazion -user:evazion`.
  Filtering out the user's own uploads and approvals was slow and usually unnecessary,
  since for most users it's rare for their own uploads or approvals to reenter the modqueue.

Before status:modqueue did this:

   SELECT * FROM posts WHERE is_pending = TRUE OR is_flagged = TRUE OR (is_deleted = TRUE AND id IN (SELECT post_id FROM post_appeals WHERE status = 0))

Now we do this:

   SELECT * FROM posts WHERE id IN (SELECT id FROM posts WHERE is_pending = TRUE UNION ALL SELECT id FROM posts WHERE is_flagged = TRUE UNION ALL SELECT id FROM posts WHERE id IN (SELECT post_id FROM post_appeals WHERE status = 0))

Postgres had a bad time with the "pending or flagged or has a pending appeal" clause because
it didn't know that posts can only be in one state at a time, so it overestimated how many
posts would be returned and chose a seq scan. Replacing the OR with a UNION avoids this.
2022-09-28 00:29:50 -05:00
evazion e72073ca6b emails: update email templates.
* Add header with the Danbooru name and logo.
* Add footer with links to the site, the privacy policy, and the contact page.
* Add "You received this email because of X" messages to remind users why
  they received the email.
* Add basic CSS to make the design match the site.
2022-09-28 00:29:50 -05:00
evazion 331f15259a Fix #1590: id metatag: "id:A..B,C..D,E"
Support searches like the following:

* score:<0,>100 (equivalent to `score:<0 or score:>100`)
* score:5,10..15,>20 (equivalent to `score:5 or score:10..15 or score:>20`)
* id:5,10..15,20..25,30 (equivalent to `id:5 or id:10..15 or id:20..25 or id:30`)

This also works inside the `search[id]` URL parameter, and inside other numeric
URL search parameters.
2022-09-27 01:44:18 -05:00
evazion 116c1f1af8 searchable: factor out metatag value parser.
Factor out the code that parses metatag values (e.g. `score:>5`) and
search URL params (e.g. `search[score]=>5`) into a RangeParser class.

Also fix a bug where, if the `search[order]=custom` param was used
without a `search[id]` param, an exception would be raised. Fix another
bug where if an invalid `search[id]` was provided, then the custom order
would be ignored and the search would be returned with the default order
instead. Now if you use `search[order]=custom` without a valid
`search[id]` param, the search will return no results.
2022-09-26 22:50:45 -05:00
evazion 65dbd89e25 Fix #4978: -approver:username implicitly adds approver:any
Fix the approver:, parent:, and pixiv: metatags not working correctly when negated:

* Fix -approver:<name> not including posts that don't have an approver (the approver_id is NULL)
* Fix -parent:<id> not including posts that don't have a parent (the parent_id is NULL)
* Fix -pixiv:<id> not including posts that aren't from Pixiv (the pixiv_id is NULL)

The problem lies how the equality operator is negated when the column contains NULL values;
`approver_id != 52664` doesn't match posts where the `approver_id` is NULL.

The search `approver:evazion` boils down to:

    # Post.where(approver_id: 52664).to_sql
    SELECT * FROM posts WHERE approver_id = 52664;

When that is negated with `-approver:evazion`, it becomes:

    # Post.where(approver_id: 52664).invert_where.to_sql
    SELECT * FROM posts WHERE approver_id != 52664;

But in SQL, `approver_id != 52664` doesn't match when the approver_id IS NULL, so the search doesn't
include posts without an approver.

We could use `a IS NOT DISTINCT FROM b` instead of `a = b`:

    # Post.where(Post.arel_table[:approver_id].is_not_distinct_from(52664)).to_sql
    SELECT * FROM posts WHERE approver_id IS NOT DISTINCT FROM 52664;

This way when it's inverted it becomes `IS DISTINCT FROM`:

    # Post.where(Post.arel_table[:approver_id].is_not_distinct_from(52664)).invert_where.to_sql
    SELECT * FROM posts WHERE approver_id IS NOT DISTINCT FROM 52664;

`approver_id IS DISTINCT FROM 52664` is like `approver_id != 52664`, except it matches when
approver_id is NULL [1].

This works correctly, however the problem is that `IS NOT DISTINCT FROM` can't use indexes because
of a long-standing Postgres limitation [2]. This makes searches too slow. So instead we do this:

    # Post.where(approver_id: 52664).where.not(approver_id: nil).to_sql
    SELECT * FROM posts WHERE approver_id = 52664 AND approver_id IS NOT NULL;

That way when negated it becomes:

    # Post.where(approver_id: 52664).where.not(approver_id: nil).invert_where.to_sql
    SELECT * FROM posts WHERE approver_id != 52664 OR approver_id IS NULL;

Which is the correct behavior.

[1] https://modern-sql.com/feature/is-distinct-from
[2] https://www.postgresql.org/message-id/6FC83909-5DB1-420F-9191-DBE533A3CEDE@excoventures.com
2022-09-26 19:28:10 -05:00
evazion a399ab1060 comments: don't nest search form under div.list-of-comments. 2022-09-26 14:41:27 -05:00
evazion 67c992bfbf post events: show post-related mod actions on post event page.
Show the following actions on the post events page:

* Post bans and unbans
* Post deletions and undeletions
* Thumbnail regenerations and IQDB regenerations
* Favorites moves
* Rating locks and unlocks
* Note locks and unlocks

Fixes #3825: Events/Moderation page for each post should show eventual ban actions
2022-09-26 03:24:50 -05:00
evazion 75a2814f18 mod actions: fix ip unban and modreport actions being visible to non-mods.
Fix IP unban actions and moderation report handled/rejected actions
being visible to non-moderators in the mod actions log.

Moderation report actions didn't leak the modreport itself, but it did
leak which moderator handled or rejected it.
2022-09-25 23:16:21 -05:00
evazion 17c6a2d77b mod actions: add options to filter /mod_actions by subject. 2022-09-25 23:02:46 -05:00
evazion 3dc765ca9d mod actions: add fix script to populate subject field.
Add a fix script to populate the mod_actions subject field by parsing
mod action descriptions. Most mod actions contain an ID, so finding the
subject is easy, but some don't. And some mod actions refer to deleted
objects, such as deleted posts or comments. In these cases the subject
will be null.

For IP bans, the mod action description only contains the IP, but it's
possible to have multiple bans for the same IP. So we look for IP bans
created by the same user, for the same IP, within the same time range.

For user bans, the mod action only contains the banned user's name and
the ban reason. This makes it difficult to find the banned user's ID in
some cases, because it's possible for the user to have changed their
name, and for the name change to have not been recorded, and for the
banner to have edited the ban reason, or for the ban to have been
deleted. So we try multiple things until we find the closest match.
2022-09-25 21:19:43 -05:00
evazion c154ce64fc modreports: fix exception when searching by recipient_name. 2022-09-25 15:06:08 -05:00
evazion 34057b25e1 mod actions: record the subject of the mod action.
Add a polymorphic `subject` field that records the subject of the mod
action. The subject is the post, user, comment, artist, etc the mod
action is for.

* The subject for the user ban and unban actions is the user, not the ban itself.
* The subject for the user feedback update and deletion actions is the user,
  not the feedback itself.
* The subject for the post undeletion action is the post, not the approval itself.
* The subject for the move favorites action is the source post where the
  favorites were moved from, not the destination post where the favorites
  were moved to.
* The subject for the post permanent delete action is nil, because the
  post itself is hard deleted.
* When a post is permanently deleted, all mod actions related to the
  post are deleted as well.
2022-09-25 04:04:28 -05:00
evazion 9026875776 db: add subject_id, subject_type columns to mod_actions.
The subject of the mod action is the post, user, or other object the modaction is for.
2022-09-25 00:03:52 -05:00
evazion 718c4d121b posts: fix double deletion bug.
Fix a bug where, if a user a was deleting a post and they accidentally
clicked the delete button twice, it could create two flags.
2022-09-24 23:41:14 -05:00
evazion 361af6a4cb posts: rework post events page.
* Add a global /post_events page that shows the history of all approvals,
  disapprovals, flags, appeals, and replacements on a single page.

* Redesign the /posts/:id/events page to show all approval, disapproval,
  flag, appeal, and replacement events for a single post (before it only
  showed approvals, flags, and appeals).

* Remove the replacement history link from the post show page. Replacements
  are now included in the post events page (closes #4948: Highlighed replacements).

* Add /post_approvals/:id and /post_replacements/:id routes (these are
  used by the "Details" link on the post events page).
2022-09-24 20:12:41 -05:00
evazion fc122cbc5a tests: fix broken tests. 2022-09-24 03:49:10 -05:00
evazion adba70a0de api: make IP addresses in the API.
Make the following fields visible in API responses:

* ip_bans.ip_addr
* ip_geolocations.ip_addr
* ip_geolocations.network
* users.last_ip_addr (mod only)
* user_sessions.ip_addr
* api_keys.last_ip_address
* api_keys.permitted_ip_addresses

Before IP addresses were globally hidden in API responses because IPs were
present in a lot of tables and we didn't want to accidentally leak them.
Now that we've gotten rid of IPs from most tables, it's safe to unhide them.
2022-09-24 03:48:45 -05:00
evazion 7bf824f0dd disapprovals: fix searching by user.
Fix searching post disapprovals by user to use the new `visible_for_search`
mechanism. This fixes it so you can do subsearches on the user field, like this:

* https://danbooru.donmai.us/post_disapprovals?search[user][level]=32 (find disapprovals by Builder-level approvers)
2022-09-23 21:17:16 -05:00
evazion 9363658652 post versions: fix exception when added_tags contains nulls.
Fix exception when viewing old post versions that contain null values in the
added_tags field: https://danbooru.donmai.us/post_versions?search[id]=1319677

There are 82 post versions with nulls in the added_tags fields and 26
with nulls in the removed_tags field.
2022-09-23 20:12:59 -05:00
evazion e5edd79180 flags: fix flagger:<name> not returning self-flagged uploads
Fix the search `flagger:evazion user:evazion` not returning the user's own self-flagged uploads.

Followup to a6e0872ce.

Fixes #4690: user profile 'flags' count links to /post_flags with different search criteria
2022-09-22 23:07:53 -05:00
evazion dff27e3a3a comments: put search form on same page as search results.
* Remove the /comment/search page. Instead put the comment search form
  on the same page as the search results, so you don't have to go back
  and forth to update your search.
* Add an "Edited?" search option, for finding comments that have been edited.
* Add options for sorting by oldest comments and lowest scoring comments.
2022-09-22 20:56:10 -05:00
evazion 0dd57a933b db: add indexes on comment fields. 2022-09-22 20:17:57 -05:00
evazion 2429d6c961 comments: let users see their own name on their own deleted comments.
* Let the user see their own username when viewing their own deleted comments.
* Don't hide the creator_id field from the comment creator in the API.
* Hide the score, do_not_bump_post, and is_sticky fields for deleted
  comments in the HTML and in the API, unless the user is a moderator.
* Hide the "..." popup menu on deleted comments, unless the user is a moderator.

This is so that when a user looks at their own comment history, their
name isn't hidden from them on their own deleted comments. This may confuse
users however into thinking their name is still visible to other users.
2022-09-22 20:01:17 -05:00
evazion a442658f8a Fix #5237: Deleted comments can be viewed by other users
* Fix it so non-moderators can't search deleted comments using the
  `updater`, `body`, `score`, `do_not_bump_post`, or `is_sticky` fields.
  Searching for these fields will exclude deleted comments.

* Fix it so non-moderators can search for their own deleted comments using the
  `creator` field, but not for deleted comments belonging to other users.

* Fix it so that if a regular user searches `commenter:<username>`, they
  can only see posts with undeleted comments by that user. If a moderator or
  the commenter themselves searches `commenter:<username>`, they can see all
  posts the user has commented on, including posts with deleted comments.

* Fix it so the comment count on user profiles only counts visible
  comments. Regular users can only see the number of undeleted comments
  a user has, while moderators and the commenter themselves can see the
  total number of comments.

Known issue:

* It's still possible to order deleted comments by score, which can let
  you infer the score of deleted comments.
2022-09-22 19:17:33 -05:00
evazion 88ac91f5f3 search: refactor to pass in the current user explicitly. 2022-09-22 04:31:21 -05:00
evazion b56b6c554b commentaries: perform full-text search instead of substring search.
Switch `/artist_commentaries?search[text_matches]` to do a full-text
search instead of a substring search. This is more consistent with text
search in other places.
2022-09-22 01:56:14 -05:00
evazion 29a4ca0818 pools: switch from search[name_matches] to search[name_contains].
The previous commit changed it so that `/pools?search[name_matches]`
does a full-text search. So for example, `search[name_matches]=smiling`
will now match pool names containing any of the words "smiling",
"smile", "smiles", or "smiled".

This commit adds a `/pools?search[name_contains]` param that does what
`name_matches` did before, and switches to it in search forms. So for
example, `search[name_contains]=smiling` will only match pool names
containing the exact substring "smiling".

This change is so that `<field>_matches` works consistently across the
site, and so that it's possible to search pool names by either an exact
substring match, or by a looser natural language match.

This is a minor breaking API change. API users can replace
`/pools?search[name_matches]` with `/pools?search[name_contains]` to get
the same behavior as before. The same applies to /favorite_groups.
2022-09-22 01:52:13 -05:00
evazion 3114ef3daf searchable: standardize the <field>_matches operator for text fields.
Standardize it so that all fields of type `text` are searchable with
`search[<field>_matches]`.

Before, the `<field>_matches` param was handled manually and some fields
were left out or handled inconsistently. Now it applies to all columns
of type `text`.

This does a full-text search on the field, so for example, searching
`/artist_commentaries?search[translated_description_matches]=smiling`
will match translated commentaries containing either the word "smiling",
"smiles", "smiled", or "smile".

Note that this only applies to columns defined as type `text`, not to
columns defined as `character varying`. The difference is that `text` is
used for fields containing free-form natural language, such as comments,
notes, forum posts, wiki pages, pool descriptions, etc, while `character
varying` is used for short strings not containing free-form language,
such as tag names, wiki page titles, urls, status fields, etc.

API changes:

* Add the `search[original_title_matches]`, `search[original_description_matches]`,
  `search[translated_title_matches]`, `search[translated_description_matches]` params
  to /artist_commentaries and /artist_commentary_versions.
* Remove the `search[name_matches]` and `search[group_name_matches]` params from /artist_versions.
* Remove the `search[title_matches]` param from /wiki_page_versions.
* Change the `search[name_matches]` param on /pools, /favorite_groups, and /pool_versions
  to do a full-text search instead of a substring match.
2022-09-22 01:52:13 -05:00
evazion a6e0872ce4 flags: fix mods not being able to see the flagger on self-flagged posts.
Fix a bug where, when a mod searched for their own flags, they couldn't
see their own self-flagged uploads.

Fix a bug where, when a mod viewed their own flags, they couldn't see
the flagger name for their own self-flagged uploads.

This also makes it so you can do things like `/post_flags?search[creator][level]=20`
to search flags by user level.
2022-09-22 01:52:13 -05:00
evazion a35f49e905 searchable: add framework for defining user search permissions.
Add a `visible_for_search` method to ApplicationPolicy that lets us
define which fields a user is allowed to search for.

For example, when a normal user searches for post flags by flagger name,
they're only allowed to see their own flags, not flags by other users.
But when a mod searches for flags by flagger name, they're allowed to
see all flags, except for flags on their own uploads.

This framework lets us define these rules in the `visible_for_search`
method in the model's policy class, rather than as special cases in the
`search` method of each model.
2022-09-22 01:51:23 -05:00
evazion 6a9a679149 searchable: refactor search_attributes helper methods into class.
Move the `search_attributes` helper methods inside the Searchable
concern into a SearchContext helper class. This is so we don't have to
pass `params` and `current_user` down through a bunch of different
helper methods, and so that these private helper methods don't pollute
the object's namespace.
2022-09-22 01:48:21 -05:00
evazion 91fca27126 db: fix types of various string columns.
Fix various columns to be either `character varying` or `text`,
depending on what kind of text is stored in the column. `text` is used
for columns that contain free-form natural language, like pool and forum
topic titles, while `character varying` is used for short strings that
don't contain free-form text, like URLs and status fields.

Both types are treated the same by Postgres; the only difference is how
we treat them in Rails. In edit forms, `text` fields use multi-line
textboxes, while `character varying` fields use single-line inputs. And
during search, we allow `text` fields to be searched using full-text
search, but not `character varying` fields.
2022-09-22 01:48:21 -05:00
evazion d730bd5882 uploads: remove unused search form template. 2022-09-22 01:48:12 -05:00
evazion 50cb30215a modqueue: add ai-generated to list of warning tags. 2022-09-22 01:48:12 -05:00
evazion 84ebef8c71 controllers: return 400 instead of 403 for GET requests with body.
Fix for 3184e77de. Return 400 Bad Request instead of 403 Forbidden for
GET requests with a body.
2022-09-22 01:48:12 -05:00
evazion a229a6f5c4 models: remove ignored_columns declarations.
These columns have been removed from the underlying database.
2022-09-20 23:09:32 -05:00
evazion b66f84c0c7 db: drop normalized_url column from artist_urls.
Unused since 0d480eb832.
2022-09-20 21:26:47 -05:00
evazion 2282fd7ea4 db: drop {updater,creator,uploader}_ip_addr columns.
Drop the updater_ip_addr, creator_ip_addr, and uploader_ip_addr columns
from various tables.
2022-09-20 20:58:10 -05:00
evazion 3184e77de0 controllers: don't allow GET requests with params in the body.
Don't allow GET requests to pass the request params in the body instead
of in the URL. While Rails can handle GET params passed in the body, it
goes against spec and it may cause problems if the response is a redirect
and the client doesn't send the body params when following the redirect.

This may be a breaking change for broken API clients who were sending
GET params in the body instead of in the URL. This can happen when people
use HTTP libraries incorrectly.
2022-09-20 20:57:49 -05:00
evazion 7977572865 users: fix user deletion validation. 2022-09-19 05:50:59 -05:00
evazion aea3837f9a users: delete accounts with invalid names.
Add a fix script to delete all accounts with invalid usernames. Also
change it so the owner-level user can delete accounts belonging to other
users.

Users who have logged in in the last year and who have a valid email
address will be given a one week warning. After that all accounts with
invalid names will be deleted. Anyone who has visited the site in the
last 6 months will have already seen a warning page that their name must
be changed to keep using the site.
2022-09-19 05:09:44 -05:00
evazion 23f9a1af7e mod actions: update /mod_actions index.
* Add newest and oldest order options.
* Rearrange columns to match /user_actions page.
2022-09-19 05:09:06 -05:00
evazion f55aa85c68 db: add category and description indexes on mod_actions table. 2022-09-18 23:31:30 -05:00
evazion 2119a8efc5 mod actions: fix messages to use consistent format.
Fix mod actions to use the same message format everywhere.

Before mod actions were formatted in various inconsistent ways:

* "deleted post #1234"
* "comment #1234 updated by <user>"
* "<user> updated forum #1234"
* "<user> level changed Member -> Builder"

Now all mod actions consistently use this format:

* "deleted post #1234"
* "updated comment #1234"
* "updated forum #1234"
* "promoted <user> from Member to Builder"

This way mod actions are formatted consistently with other actions on
the /user_actions page, where everything is written as "<user> did X".

Also add a fix script to fix existing mod actions.
2022-09-18 21:56:57 -05:00
evazion 72e95b6ca3 flags: allow approvers to bypass the "can't flag more than once in 3 days" rule.
Allow approvers to bypass the rule that you can't flag a post again if
it was flagged less than 3 days ago. This rule was intended to prevent
flag warring among regular users, which hopefully shouldn't be a problem
among approvers. It was also useless because approvers could always
just directly delete the post even if they couldn't flag it.

Allowing approvers to reflag posts allows them to reinstate flags that
were accidentally approved.
2022-09-18 15:56:35 -05:00
evazion 0c919a6bc8 versions: remove 'subsequent' version comparison option.
This option was rarely used and what it actually did was usually
difficult to understand.
2022-09-18 15:56:10 -05:00
evazion 1d2bac7b95 Remove CurrentUser.ip_addr.
Remove the `CurrentUser.ip_addr` global variable and replace it with
`request.remote_ip`. Before we had to track the current user's IP in a
global variable so that when we edited a post for example, we could pass
down the user's IP to the model and save it in the post_versions table.
Now that we now longer save IPs in version tables, we don't need a global
variable to get access to the current user's IP outside of controllers.
2022-09-18 05:02:10 -05:00
evazion d4da8499ce models: stop saving IP addresses in version tables.
Mark various `creator_ip_addr` and `updater_ip_addr` columns as ignored
and stop updating them in preparation for dropping them.
2022-09-18 03:49:17 -05:00
evazion 44d4452068 db: set various ip address columns to nullable.
Set various IP address columns to nullable in preparation for dropping them.

In production, some of these tables already contained null values even
though it violated the constraint.
2022-09-17 23:30:13 -05:00
evazion 553d35178c Remove IpAddress model. 2022-09-17 23:30:13 -05:00
evazion 075199cd1e Remove /ip_addresses page.
Remove the /ip_addresses page. This page allowed moderators to search
users by IP, and to see recent activity tied to an IP. However, it was
limited to IPs tied to uploads, comments, dmails, artist edits, note
edits, and wiki edits.

Remove this page because it was limited in scope and because there are
better ways of doing what it did. The /user_events page is better at
catching sockpuppets because it tracks IPs for every login, not just for
certain types of edits. And the /user_actions page is better at
monitoring user activity because it shows all activity associated with
an account, not just for certain types of edits.

Removing this allows us to drop IP addresses from all tables besides the
user_events table. This is good because these IPs are no longer necessary
for any purpose, and because storing them forever is a liability.
2022-09-17 21:32:26 -05:00
evazion 1ac56b72a7 emails: add proton.me to domain whitelist. 2022-09-16 06:01:44 -05:00
evazion 1e3fd3b724 artist finder: add pomf.tv to site blacklist. 2022-09-16 06:01:44 -05:00
evazion 3d092bfd37 site map: reorganize layout.
Reorganize the /static/site_map page to try to group things together in
a more logical fashion, especially the Admin section.
2022-09-16 06:01:44 -05:00
evazion a62e844a1a forum: fix visibility of forum post votes.
Make all forum post votes visible to everyone.

When forum votes were first introduced, it was technically possible to
vote on any forum post, including on posts in mod-only threads.
Accordingly, forum post votes were only visible if the forum post itself
was visible. However, there doesn't actually exist any votes on private
forum posts, and trying to filter them out makes the /user_actions page
much slower, so just make them visible to everyone.
2022-09-16 06:01:44 -05:00
evazion bd73090b4c user events: make all events visible to moderators.
Allow moderators to see all events on the /user_events page. Before only
admins could see when a user changed their email, changed their
password, or had a failed login attempt. Now moderators can see these
events too.

Filtering these events out made the /user_actions page slower, and it
wasn't really necessary since merely knowing that a user changed their
email or password isn't that much more sensitive than knowing when they
logged in or out.
2022-09-16 06:01:44 -05:00
evazion ee638f976f Add /user_actions page.
Add a /user_actions page. This page shows you a global timeline of
(almost) all activity on the site, including uploads, comments, votes,
edits, forum posts, and so on.

The main things it doesn't include are post edits, pool edits, and
favorites (posts and pools live in a separate database, and favorites
don't have the timestamps we need for ordering).

This page is useful for moderation purposes because it lets you see a
history of almost all of a user's activity on a single page.

Currently this page is mod-only. In the future it will be open to all
users, so you can view the history of your own site activity, or the
activity of others.
2022-09-16 05:39:25 -05:00
evazion 0830af49a7 db: add user_actions view.
Add a user_actions view. This view unions together a bunch of tables to
produce an event log of every action taken by a user.

Also add a bunch of indexes to make queries on this table efficient.
Even though the view is an enormous query combining together about 30
different tables, queries are very efficient as long as every table has
`created_at` and `(user_id, created)` indexes.
2022-09-16 04:20:19 -05:00
evazion abf493794f twitter: fix misparsing of https://twitter.com/i/status/:id urls.
Fix URLs like `https://twitter.com/i/status/943446161586733056` parsing
the username as `i`. This led to the new artist page recommending the
tag name `i` when creating an artist for a source like this.

Also fix these URLs not being normalized to `https://twitter.com/:username/status/:id` after upload.
2022-09-15 19:57:12 -05:00
evazion cfe567b649 uploads: fix exception in UploadMediaAsset.visible.
Fix `UploadMediaAsset.visible(user).count` failing when we weren't
joined on the uploads table.
2022-09-15 19:19:44 -05:00
evazion 0a5ebcc69d uploads: refactor media asset validation logic.
Refactor the upload validation logic to not depend on the current user.
Fixes several broken upload tests.
2022-09-15 05:09:07 -05:00
evazion bfe2eabc6d db: change ids from bigint to integer on various tables.
Change ID columns from `bigint` (64-bits) to `integer` (32-bits) on various tables.

Rails 6.0 switched the default from bigint to integer for IDs on new
tables, so now we have a mix of tables with integer IDs and bigint IDs.
Switch back to integer IDs on certain tables because we're going to
build a view that unions a bunch of tables together to build a user
activity timeline, and for this purpose all the tables need to have IDs
of the same type in order for Postgres to optimize the query effectively.
2022-09-15 03:47:05 -05:00
evazion 86b2d25f21 views: fix search forms to retain limit param.
Fix search forms to retain the `limit` URL param if it was manually given in the URL.
2022-09-15 03:47:05 -05:00
evazion 25739d9e29 Merge pull request #5226 from kidonng/favorite-groups
Add keyboard shortcuts for favorite groups
2022-09-15 03:46:50 -05:00
evazion 9e16de13ef Merge pull request #5220 from nonamethanks/duration-validation
Uploads: allow admins to bypass duration limits again
2022-09-15 03:46:21 -05:00
evazion 04f98d3b8c Merge pull request #5246 from NamelessContributor/fix-long-tag-overflow
Fix some long tags / wiki links overflowing their container
2022-09-15 03:44:15 -05:00
evazion ab900beffd Merge pull request #5250 from jwood7423/Add-api-to-the-list-of-meta-wikis
Added `api:` to the list of META_WIKIS
2022-09-15 03:43:42 -05:00
evazion f504f752d0 Merge pull request #5243 from nonamethanks/remove-self-upload-from-highlights
Modqueue: remove self_upload from highlighted tags
2022-09-15 03:42:17 -05:00
evazion e2a3265daf mod dashboard: remove ip address search.
Remove the IP address search option from the /moderator/dashboard page.
This was an obsolete way of searching for sockpuppet accounts by IP.
The /user_events page should be used instead.
2022-09-13 00:14:19 -05:00
evazion fb980d4a16 notes: merge versions when note is deleted.
Unlike other models, notes had a special rule where if you deleted or
undeleted a note, it would always create a new version instead of
merging it into the previous version. Remove this rule since it didn't
have a purpose and it was inconsistent with other versioned models.
2022-09-12 22:10:14 -05:00
evazion 0267e4e3d7 wikis: fix exception when wiki doesn't have a tag.
Fix a nil reference exception in the sidebar when a wiki page doesn't
belong to a tag.

Also hide the options sidebar on the new wiki page since none of the
options are relevant when creating a new wiki.
2022-09-12 21:24:03 -05:00
evazion e1fb219738 aliases: retire artist aliases after 2 years.
Automatically retire artist aliases after they're more than 2 years old.
Before the rule was that artist aliases were only retired if they didn't
have any new posts in the last 2 years.
2022-09-12 02:31:12 -05:00
evazion 2c37fdf9e8 tags: don't create mod action when tag is deprecated.
Not needed anymore since deprecations are now tracked in the tag history.
2022-09-12 02:06:16 -05:00
evazion 600e75e324 /tags: update tag actions menu.
Replace the "Edit | History | Related | Similar" links on the /tags
index with a popup menu containing more actions, including links to the
tag's version history, links to the tag's aliases and implications,
links to comments and forum discussions mentioning the tag, and links to
find untagged and mistagged posts using AI tags.
2022-09-12 00:28:26 -05:00
evazion bb728ecebf tags: add /tag_versions page. 2022-09-11 18:41:16 -05:00
evazion 54a45a3021 tags: track tag histories.
Track the history of the tag `category` and `is_deprecated` fields in
the `tag_versions` table.

Adds generic Versionable and VersionFor concerns that encapsulate most
of the history tracking logic. These concerns are designed to make it
easy to add history to any model.

There are a couple notable differences between tag versions and other versions:

* There is no 1 hour edit merge window. All changes to the `category`
  and `is_deprecated` fields produce a new version in the tag history.

* New versions aren't created when a tag is created. Versions are only
  created when a tag is edited for the first time. The tag's initial
  version isn't created until *after* the tag is edited for the first time.

For example, if you change the category of a tag that was last updated
10 years ago, that will create an initial version of the tag backdated
to 10 years ago, plus a new version for your edit.

This is for a few reasons:

* So that we don't have to create new tag versions every time a new tag
  is created. This would be wasteful because most tags never have their
  category or deprecation status change.
* So that if you make a typo tag, your name isn't recorded in the tag's
  history forever.
* So that we can create new tags in various places without having to know
  who created the tag (which may be unknown if the current user isn't set).
* Because we don't know the full history of most tags, so we have to
  deal with incomplete histories anyway.

This has a few important consequences:

* Most tags won't have any tag versions. They only gain tag versions if
  they're edited.
* You can't track /tag_versions to see newly created tags. It only
  shows changes to already existing tags.
* Tag version IDs won't be in strict chronological order. Higher IDs may
  have created_at timestamps before lower IDs. For example, if you
  change the category of a tag that is 10 years old, that will create an
  initial version with a high ID, but with a created_at timestamp dated
  to 10 years ago.

Fixes #4402: Track tag category changes
2022-09-11 17:47:44 -05:00
evazion 0c327a2228 tags: add tag_versions table.
Add a tag_versions table for tracking the history of tags.

A couple notable differences from other version tables:

* There is a previous_version_id column that points to the previous
  version. This allows finding the first, last, previous, or next
  version efficiently in SQL.

* There is a `version` column that tracks the revision number (1, 2, 3, etc).
  Post versions and note versions have this, but other version tables don't.

* The `updater_id` column is optional. This is because we don't know who
  the last updater was before we started tracking the history of tags,
  so the initial updater will be NULL in the first version of the tag.
2022-09-11 17:35:53 -05:00
evazion 10cb97dbd5 Fix #5200: non-web_source, bad_source, etc. not removed when using source: metatag 2022-09-11 03:03:57 -05:00
evazion 2eead46ad4 tags: remove dead code. 2022-09-10 14:39:17 -05:00
evazion 22bfa44183 posts: fix exception when tagging post with char:copy:foo.
Fixup for 015c6dc7d. Show a warning about failure to add a tag instead
of raising an exception when trying to tag a post with `char:copy:foo`.
This tries to create a tag named `copy:foo` then set the category to
character, which doesn't work because `copy:foo` isn't a valid tag name.
2022-09-10 14:39:17 -05:00
evazion 015c6dc7db Fix #4965: Account for metatag prefixes when searching/linking
Drop the ability to write e.g. `create alias foo -> char:bar` in a BUR
to change the tag's type as a side effect. You can only use these
tag type prefixes in tag edits now.

This feature was only intended to be used in tag edits. The fact it
worked elsewhere was unintended behavior.

This feature was problematic because it relied on `Tag.find_or_create_by_name`
automagically changing the tag's category when the tag name contained a
tag category prefix, e.g. `char:hatsune_miku`. This meant that merely
looking up a tag could have the side effect of changing its category.
It was also bad because `find_or_create_by_name` had a hidden dependency
on the current user, which may not be set or available in all contexts.
2022-09-10 04:49:24 -05:00
evazion f36f1ff37b tags: drop is_locked column.
This column was deprecated in 208b6189. Finish removing it.
2022-09-09 15:58:48 -05:00
evazion c96bdd1766 autocomplete: fix ranking of exact matches.
Fix a bug where searching for `sakana~` ranked `sakana~_(meme)` beneath
random artist tags containing the word `sakana`. Now, if the search contains
punctuation, we rank exact matches first, even for small tags. Before we
ranked exact matches for small tags lower than inexact matches for large
tags. If the search contains punctuation, it's a strong signal the user
is looking for an exact match.
2022-09-09 15:58:48 -05:00
evazion 0cc76625eb Update Ruby gems and Yarn packages. 2022-09-07 03:13:13 -05:00
evazion a80ee22746 autocomplete: fix incorrect highlighting when tag contains repeated words.
Fix tags like `short_shorts` or `hunter_x_hunter` being highlighted
incorrectly. Typing `short_sh` would highlight it as SHort_SHorts
instead of as SHORT_SHorts.
2022-09-06 18:01:30 -05:00
evazion d2147eca80 tumblr: fix exception when fetching data for video urls.
Fix an exception when trying to fetch source data for URLs like
https://va.media.tumblr.com/tumblr_pgohk0TjhS1u7mrsl.mp4.

For these URLs it's not possible to use the trick where we try to open
the URL as a HTML page and scrape the post id from the HTML. Instead we
get the raw video if we try to to this.
2022-09-05 16:15:47 -05:00
evazion f55951ab58 tumblr: fix exception when parsing mangled image urls.
Fix a nil exception when trying to parse invalid URLs like `https://25.media.tumblr.com/91719d337b218681abc48cdc24e`.
2022-09-05 16:15:46 -05:00
evazion 3d5b201e0d tests: fix upload tests. 2022-09-03 23:54:28 -05:00
evazion 1968c9c69d autocomplete: lowercase searches clientside.
Lowercase searches clientside before sending them to the server. This is
to improve cache hit rates when users accidentally type in uppercase,
especially on mobile where the first letter often gets capitalized.
2022-09-03 23:54:28 -05:00
evazion 306f3d8f1a autocomplete: add tests. 2022-09-03 20:34:34 -05:00
evazion 4a77d67d1f autocomplete: fix completion of ai: metatag.
Fix it so we don't show the `ai:` prefix in front of results in the
autocomplete menu when completing the `ai: metatag.
2022-09-03 20:34:34 -05:00
evazion f83af31a00 autocomplete: fix usernames not being highlighted in @mentions.
Fix usernames not being highlighted when completing @mentions.

This also changes it so the autocomplete results don't include the '@'
in front of the name.

Minor breaking change to the /autocomplete.json API. Autocomplete
results for mentions now have the type `mention` instead of `user`.
2022-09-03 20:34:26 -05:00
evazion 03c02ef78e autocomplete: fix accidental API breakage.
Fix /autocomplete.json returning results like this:

    [
      {
        "table": {
          "type": "tag-word",
          "label": "long hair",
          "value": "long_hair",
          "category": 0,
          "post_count": 2818211,
          "antecedent": null
        }
      },
    ]

instead of like this:

    [
      {
        "type": "tag-word",
        "label": "long hair",
        "value": "long_hair",
        "category": 0,
        "post_count": 2818211,
      },
    ]

Also change it so that optional attributes like `antecedent` aren't
returned if they're null. This could be a minor breaking change if users
rely on these attributes always being present.

Lastly change XML results to look like this:

    <autocomplete-service-results type="array">
      <autocomplete-service-result>
        <type>tag-word</type>
        <label>long hair</label>
        <value>long_hair</value>
        <category type="integer">0</category>
        <post-count type="integer">2801117</post-count>
      </autocomplete-service-result>
    </autocomplete-service-results>

instead of like this:

    <objects type="array">
      <object>
        <type>tag</type>
        <label>hair ornament</label>
        <value>hair_ornament</value>
        <category type="integer">0</category>
        <post-count type="integer">883340</post-count>
        <antecedent nil="true"/>
      </object>
    </objects>
2022-09-03 18:53:29 -05:00