Comments have three states: visible, hidden, and invisible. Visible
comments are always shown. Hidden comments are not shown until the user
clicks 'Show all comments'. Invisible comments are never shown to the
user. Deleted comments are treated as hidden for moderators and
invisible for normal users. Thresholded comments are treated as hidden
for all users.
Rewrite the implementation of related tags to be simpler, faster, and
more accurate:
* The related tags are now calculated by taking a random sample of 1000
posts, finding the top 250 most frequent tags among those posts, then
ordering those tags by cosine similarity.
* Related tags can generally be calculated in 50-300ms at these sample
sizes. Very high sample sizes (25000+ posts) are still relatively fast
(1-3 seconds), but generally they don't improve accuracy much.
* Related tags are now cached in redis rather than in the tags table.
The related_tags column in the tags table is no longer used.
* Only the related tags in the search taglist are cached. The related
tags returned by the 'Related tags' button are not cached.
* The cache lifetime is a fixed 4 hours.
* The 'Related tags' button now works with metatags.
* The /related_tag page now works with metatags and multitag searches.
Fixes#4134, #4146.
* Drop support for `source:pixiv/artist-name` searches. This was a hack
that only worked on old pixiv urls that haven't been used for years.
* Replace the old SourcePattern(lower(source)) index with a trigram index.
Drop support for https://danbooru.donmai.us/cache/tags.json. This was a
nightly dump of the tags table that was originally added in #1012. It
was never documented and never really used except for by the DanbooruUp
extension.
This fixes InvalidAuthenticityToken errors caused by Reportbooru trying to
use this endpoint to update related tags. Reportbooru uses a secret key
to authenticate rather than using apikey-based authentication, which
makes it fail the CSRF protection.
Bug: if all the comments on a post were deleted then the deleted
comments wouldn't be visible to moderators.
This was because we assumed that if `last_commented_at` was nil it meant
that the post had no comments, but this was wrong. `last_commented_at`
only counts undeleted comments. It's reset to nil if all the commnets
have been deleted.
Previously if a pool had >100 posts then the 'Order' link wouldn't
appear in the navbar, but it was still possible to visit the pool order
page directly. If a user did so, only the first 100 posts in the pool
would be shown in the drag-and-drop widget. If they tried to reorder the
pool anyway, then everything beyond the first 100 posts would be
silently removed from the pool.
Now we always show the 'Order' link, but we disable the drag-and-drop
reordering widget when the pool has >100 posts to prevent posts from
being silently removed.
Foreman is unmaintained and locked to a really old version of thor,
which prevents upgrading to Rails 6.
The Procfile can still be used by manually installing Foreman with
`gem install foreman`. This is what Foreman's README recommends anyway.
Refactor to use `render_error_page` to handle User::PrivilegeError
exceptions. This way these exceptions are logged to New Relic.
Changes:
* Anonymous users aren't automatically redirected to the login page.
Instead they're taken to the access denied page, which links to the
login/signup pages.
* JSON/XML error responses return `message` instead of `reason`.
* Refactor api_check to use render_error_page so that api limit errors
get logged to New Relic for analysis.
* Also standardize json error responses to return the error message in
`message` instead of `reason`.
Fixes POST/PUT API requests failing with InvalidAuthenticityToken errors
due to missing CSRF tokens.
CSRF protection is only necessary for cookie-based authentication. For
non-cookie-based authentication we can safely disable it. That is, if
the user is already passing their login + api_key, then we don't need
to additionally verify the request with a CSRF token.
ref: 2e407fa476 (comments)
