Refactor full-text search on several tables (comments, dmails,
forum_posts, forum_topics, notes, and wiki_pages) to use to_tsvector
expression indexes instead of dedicated tsvector columns. This way
full-text search works the same way across all tables.
API changes:
* Changed /wiki_pages.json?search[body_matches] to match against only
the body. Before `body_matches` matched against both the title and the body.
* Added /wiki_pages.json?search[title_or_body_matches] to match against
both the title and the body.
* Fixed /dmails.json?search[message_matches] to match against both the
title and body when doing a wildcard search. Before a wildcard search
only matched against the body.
* Added /dmails.json?search[body_matches] to match against only the dmail body.
Drop the final dependency on the Postgres test_parser extension.
We also have to remove references to test_parser in the migration where
it was first defined, otherwise replaying all migrations from the
beginning will fail. Replaying all migrations from the beginning
normally isn't done except in testing.
After this, it should be possible to use a vanilla install of Postgres
with Danbooru. It's still recommended to use Danbooru's Docker image for
Postgres (https://ghcr.io/danbooru/postgres), as other Postgres extensions
may be necessary in the future.
Change the wiki_pages tsvector_update_trigger to use
`pg_catalog.english` instead of `public.danbooru`. This changes how wiki
page text is parsed for full-text search to use the standard English
parser instead of test_parser. This is to prepare for dropping
test_parser. Using test_parser here was wrong anyway because it meant
that punctuation wasn't removed from words when indexing wiki pages for
full-text search.
Merge the 100 favorite subtables into a single table.
Previously the favorites table was partitioned by user id into 100
subtables to try to make searching by user id faster. This wasn't really
necessary and probably slower than just making an index on
(favorites.user_id, favorites.id) to satisfy ordfav searches. BTree
indexes are logarithmic so dividing an index by 100 doesn't make it 100
times faster to search; instead it just removes a layer or two from the
tree.
This also adds a uniqueness index on (user_id, post_id) to prevent
duplicate favorites. Previously we had to check for duplicates at the
application layer, which required careful locking to do it correctly.
Finally, this adds an index on favorites.id, which was surprisingly
missing before. This made ordering and deleting favorites by id really
slow because it degraded to a sequential scan.
Add a model for storing image and video metadata for uploaded files.
Metadata is extracted using ExifTool. You will need to install ExifTool
after this commit. ExifTool 12.22 is the minimum required version
because we use the `--binary` option, which was added in this release.
The MediaMetadata model is separate from the MediaAsset model because
some files contain tons of metadata, and most of it is non-essential.
The MediaAsset model represents an uploaded file and contains essential
metadata, like the file's size and type, while the MediaMetadata model
represents all the other non-essential metadata associated with a file.
Metadata is stored as a JSON column in the database.
ExifTool returns all the file's metadata, not just the EXIF metadata.
EXIF is one of several types of image metadata, hence why we call
it MediaMetadata instead of EXIFMetadata.
A MediaAsset represents an image or video file uploaded to Danbooru. It
stores the metadata associated with the image or video. This is to work
on decoupling files from posts so that images can be uploaded separately
from posts.
Make it so that when a user removes their own vote, the vote is soft
deleted (the is_deleted flag is set) instead of hard deleted.
Changes:
* Add is_deleted flag to comment votes.
* Relax uniqueness constraint so you can have multiple deleted votes on
the same comment. You can still only have one active vote on the comment.
* Add `soft_delete` method to Deletable concern.
Changes:
* Change the `expires_at` field to `duration`.
* Make moderators choose from a fixed set of standard ban lengths,
instead of allowing arbitrary ban lengths.
* List `duration` in seconds in the /bans.json API.
* Dump bans to BigQuery.
Note that some old bans have a negative duration. This is because their
expiration date was before their creation date, which is because in 2013
bans were migrated to Danbooru 2 and the original ban creation dates
were lost.
Rework the rate limit implementation to make it more flexible:
* Allow setting different rate limits for different actions. Before we
had a single rate limit for all write actions. Now different
controller endpoints can have different limits.
* Allow actions to be rate limited by user ID, by IP address, or both.
Before actions were only limited by user ID, which meant non-logged-in
actions like creating new accounts or attempting to login couldn't be rate
limited. Also, because actions were limited by user ID only, you could
use multiple accounts with the same IP to get around limits.
Other changes:
* Remove the API Limit field from user profile pages.
* Remove the `remaining_api_limit` field from the `/profile.json` endpoint.
* Rename the `X-Api-Limit` header to `X-Rate-Limit` and change it from a
number to a JSON object containing all the rate limit info
(including the refill rate, the burst factor, the cost of the call,
and the current limits).
* Fix a potential race condition where, if you flooded requests fast
enough, you could exceed the rate limit. This was because we checked
and updated the rate limit in two separate steps, which was racy;
simultaneous requests could pass the check before the update happened.
The new code uses some tricky SQL to check and update multiple limits
in a single statement.
Add the ability to restrict API keys so that they can only be used with
certain IP addresses or certain API endpoints.
Restricting your key is useful to limit damage in case it gets leaked or
stolen. For example, if your key is on a remote server and it gets
hacked, or if you accidentally check-in your key to Github.
Restricting your key's API permissions is useful if a third-party app or
script wants your key, but you don't want to give full access to your
account.
If you're an app or userscript developer, and your app needs an API key
from the user, you should only request a key with the minimum
permissions needed by your app.
If you have a privileged account, and you have scripts running under
your account, you are highly encouraged to restrict your key to limit
damage in case your key gets leaked or stolen.
Should improve performance for rating:e and rating:q searches. Rating:s
isn't isn't indexed because Postgres is unlikely to use the index for
rating:s searches (the selectivity is too low, ~77% of all posts are
rating:s).
Fix various minor inconsistencies between the production database schema
and the declared schema in db/structure.sql.
* tags.category was a smallint instead of an integer in production.
* The unique_schema_migrations index didn't exist outside production.
* The index_posts_on_tag_index index was called index_posts_on_tags_index
outside production.
* The posts.tag_index column didn't have a statistics target defined
outside production.
* ID sequences didn't have `AS integer` defined in production.
Specify the default settings for new users inside the User model instead
of inside the database. This makes it easier to change defaults, and it
makes the code clearer.
Optimize searches for non-English phrases in autocomplete. These
searches were pretty slow, and could sometimes cause sitewide lag spikes
when users typed long strings of non-English text into the search box
and caused an unintentional DoS.
The trick is to use an `array_to_tsvector(other_names) USING gin` index
on other_names. This supports fast string prefix matching against all
elements of the array. The downside is that it doesn't allow infix or
suffix matches, so we can't support wildcards in general. Wildcards
didn't quite work anyway, since artist and wiki other names can contain
literal '*' characters.
Add tracking of certain important user actions. These events include:
* Logins
* Logouts
* Failed login attempts
* Account creations
* Account deletions
* Password reset requests
* Password changes
* Email address changes
This is similar to the mod actions log, except for account activity
related to a single user.
The information tracked includes the user, the event type (login,
logout, etc), the timestamp, the user's IP address, IP geolocation
information, the user's browser user agent, and the user's session ID
from their session cookie. This information is visible to mods only.
This is done with three models. The UserEvent model tracks the event
type (login, logout, password change, etc) and the user. The UserEvent
is tied to a UserSession, which contains the user's IP address and
browser metadata. Finally, the IpGeolocation model contains the
geolocation information for IPs, including the city, country, ISP, and
whether the IP is a proxy.
This tracking will be used for a few purposes:
* Letting users view their account history, to detect things like logins
from unrecognized IPs, failed logins attempts, password changes, etc.
* Rate limiting failed login attempts.
* Detecting sockpuppet accounts using their login history.
* Detecting unauthorized account sharing.
In Danbooru 1, aliases (and implications) had a `reason` field where
either the admin or the alias requester gave a reason for the alias.
This field was removed from the code and the database schema, but it
still existed in the production database. This adds the field back, so
that the dev schema is consistent with the production schema, and so
that legacy reasons can be viewed on site again.
* Add back legacy tag_aliases.reason and tag_implications.reason field.
* Make /tag_aliases and /tag_implications show legacy reasons.
* Add the reason field to the search form.
* Remove the PostRegeneration model. Instead just use a mod action
to log when a post is regenerated.
* Change it so that IQDB is also updated when the image samples are
regenerated. This is necessary because when the images samples are
regenerated, the thumbnail may change, which means IQDB needs to be
updated too. This can happen when regenerating old images with
transparent backgrounds where the transparency was flattened to black
instead of white in the thumbnail.
* Only display one "Regenerate image" option in the post sidebar, to
regenerate both the images and IQDB. Regenerating IQDB only can be
done through the API. Having two options in the sidebar is too much
clutter, and it's too confusing for Mods who don't know the difference
between an IQDB-only regeneration and a full image regeneration.
* Add a confirm prompt to the "Regenerate image" link.
Add a model to store the status of user upgrades.
* Store the upgrade purchaser and the upgrade receiver (these are
different for a gifted upgrade, the same for a self upgrade).
* Store the upgrade type: gold, platinum, or gold-to-platinum upgrades.
* Store the upgrade status:
** pending: User is still on the Stripe checkout page, no payment
received yet.
** processing: User has completed checkout, but the checkout status in
Stripe is still 'unpaid'.
** complete: We've received notification from Stripe that the payment
has gone through and the user has been upgraded.
* Store the Stripe checkout ID, to cross-reference the upgrade record on
Danbooru with the checkout record on Stripe.
This is the upgrade flow:
* When the user clicks the upgrade button on the upgrade page, we call
POST /user_upgrades and create a pending UserUpgrade.
* We redirect the user to the checkout page on Stripe.
* When the user completes checkout on Stripe, Stripe sends us a webhook
notification at POST /webhooks/receive.
* When we receive the webhook, we check the payment status, and if it's
paid we mark the UserUpgrade as complete and upgrade the user.
* After Stripe sees that we have successfully processed the webhook,
they redirect the user to the /user_upgrades/:id page, where we show
the user their upgrade receipt.
Rename the following post replacement attributes:
* file_size_was -> old_file_size
* file_ext_was -> old_file_ext
* image_width_was -> old_image_width
* image_height_was -> old_image_height
* md5_was -> old_md5
In Rails 6.1, having attributes named `file_size` and `file_size_was` on
the same model breaks things because it conflicts with Rails' dirty
attribute tracking.
Tune autocorrect to produce fewer false positives. Before we used
trigram similarity. Now we use Levenshtein edit distance with a dynamic
typo threshold. Trigram similarity was able to correct large
transpositions (e.g. `miku_hatsune` -> `hatsune_miku`), but it was bad
at correcting small typos. Levenshtein is good at small typos, but can't
correct large transpositions.
Remove the pending status from tag aliases and implications.
Previously aliases would be created first in the pending state then
changed to active when the alias was later processed in a delayed job.
This meant that BURs weren't processed completely sequentially; first
all the aliases in a BUR would be created in one go, then later they
would be processed and set to active sequentially.
This was problematic in complex BURs that tried to reverse or swap
around aliases, since new pending aliases could be created before old
conflicting aliases were removed.
* Add index on posts.is_deleted. The modqueue was slow because we the
appeal condition wasn't constrained to deleted posts, so it degraded to
a full table scan.
* Avoid extra queries for calculating the page count and disapproval counts.
* Include appealed posts in the modqueue.
* Add `status` field to appeals. Appeals start out as `pending`, then
become `rejected` if the post isn't approved within three days. If the
post is approved, the appeal's status becomes `succeeded`.
* Add `status` field to flags. Flags start out as `pending` then become
`rejected` if the post is approved within three days. If the post
isn't approved, the flag's status becomes `succeeded`.
* Leave behind a "Unapproved in three days" dummy flag when an appeal
goes unapproved, just like when a pending post is unapproved.
* Only allow deleted posts to be appealed. Don't allow flagged posts to be appealed.
* Add `status:appealed` metatag. `status:appealed` is separate from `status:pending`.
* Include appealed posts in `status:modqueue`. Search `status:modqueue order:modqueue`
to view the modqueue as a normal search.
* Retroactively set old flags and appeals as succeeded or rejected. This
may not be correct for posts that were appealed or flagged multiple
times. This is difficult to set correctly because we don't have
approval records for old posts, so we can't tell the actual outcome of
old flags and appeals.
* Deprecate the `is_resolved` field on post flags. A resolved flag is a
flag that isn't pending.
* Known bug: appealed posts have a black border instead of a blue
border. Checking whether a post has been appealed would require either
an extra query on the posts/index page, or an is_appealed flag on
posts, neither of which are very desirable.
* Known bug: you can't use `status:appealed` in blacklists, for the same
reason as above.
Several fixes for the "This tag is under discussion" notice on the post
index page:
* Fix the notice appearing for BURs that aren't pending.
* Fix the notice never going away because of the cache never expiring.
* List all topics when a tag is involved in multiple BURs.
* Link to the forum post instead of the forum topic (fix#4421).
* Optimization: don't check for BURs when the search isn't a simple
single tag search.
* Add a `tags` field to the bulk update requests table for tracking all
tags involved in the request (excluding tags in mass updates that are
negated/optional/wildcards). Known issue: doesn't handle tag type
prefixes in mass updates correctly (e.g. `mass update foo -> artist:bar`
doesn't detect the tag `bar`).
* Allow searching the /bulk_update_requests page by tags.
We don't really need to cache the notice here, but we do it anyway to
reduce queries on the post index page.
* Make IP bans soft deletable.
* Add a hit counter to track how many times an IP ban has blocked someone.
* Add a last hit timestamp to track when the IP ban last blocked someone.
* Add a new type of IP ban, the signup ban. Signup bans restrict new
signups from editing anything until they've verified their email
address.
* Move emails from users table to email_addresses table.
* Validate that addresses are formatted correctly and are unique across
users. Existing invalid emails are grandfathered in.
* Add is_verified flag (the address has been confirmed by the user).
* Add is_deliverable flag (an undeliverable address is an address that bounces).
* Normalize addresses to prevent registering multiple accounts with the
same email address (using tricks like Gmail's plus addressing).
The old password reset flow:
* User requests a password reset.
* Danbooru generates a password reset nonce.
* Danbooru emails user a password reset confirmation link.
* User follows link to password reset confirmation page.
* The link contains a nonce authenticating the user.
* User confirms password reset.
* Danbooru resets user's password to a random string.
* Danbooru emails user their new password in plaintext.
The new password reset flow:
* User requests a password reset.
* Danbooru emails user a password reset link.
* User follows link to password edit page.
* The link contains a signed_user_id param authenticating the user.
* User changes their own password.
Rename is_active to is_deleted. This is for better consistency with
other models, and to reduce confusion over what "active" means for
artists. Sometimes users think active is for whether the artist is
actively producing work.
