* Swap out activerecord-hierarchical_query gem for some guy's patched
version because the mainline version is incompatible with 6.1.
* Disable meta_request gem because it hangs puma on startup on 6.1.
Expand the tag abbreviation system introduced in b0be8ae45 so that it
works in searches and when tagging posts, not just in autocomplete.
For example, you can tag a post with /evth and it will add the tag
eyebrows_visible_through_hair. You can search for /evth and it will
search for the tag eyebrows_visible_through_hair.
Some more examples:
* /ops is short for one-piece_swimsuit
* /hooe is short for hair_over_one_eye
* /saol is short for standing_on_one_leg
* /tlozbotw is short for the_legend_of_zelda:_breath_of_the_wild
If two tags have the same abbreviation, then the larger tag takes
precedence. For example, /be is short for blue_eyes, not brown_eyes,
because blue_eyes is the bigger tag.
If there is an existing shortcut alias that conflicts with the
abbreviation, then the alias take precedence. For example, /sh is short
for suzumiya_haruhi, not short_hair, because there's an old alias for
/sh -> suzumiya_haruhi.
Limit tag length to 170 chars. 170 chars was chosen because it's
longer than the longest active tag on Danbooru.
Tag length is limited because in some contexts we can't deal with
excessively long tags. Tag autocorrect for example uses the levenshtein
function in Postgres, which can't handle strings more than 255 chars long.
Disallow tags from starting with a '/' character. This is so that tag
abbreviations in autocomplete, which start with a '/', don't conflict
with regular tags.
Also disallow some other punctuation characters: `%{})]. Currently no
tags start with these characters. This is to reserve other special
characters in case we need them for other future syntax extensions.
Add the following search operators:
* /tags?search[post_count_eq]=42
* /tags?search[post_count_not_eq]=42
* /tags?search[post_count_gt]=42
* /tags?search[post_count_gteq]=42
* /tags?search[post_count_lt]=42
* /tags?search[post_count_lteq]=42
Works for all numeric attributes on all index actions.
This is the scenario:
* You type something in autocomplete, let's say 'touhou'.
* Autocomplete calls /autocomplete?search[query]=touhou&search[type]=tag_query
* The endpoint returns JSON, because the autocomplete call sets an
`Accept: application/json` header requesting JSON.
* Visit /autocomplete?search[query]=touhou&search[type]=tag_query in your browser.
* Notice that the cached JSON response is incorrectly returned, not an
HTML response like the browser requested.
The problem is that the response type is chosen based on the Accept
header, but the response didn't set the `Vary: Accept` header, so the
browser doesn't know the response type can vary and so it incorrectly
returns the cached response.
This issue is partially fixed by Rails 6.1 ([1]), which properly sets the
`Vary: Accept` header when the response depends on the Accept header.
However, the next issue is that Cloudflare doesn't respect the Vary
header at all ([2], [3]). Therefore we can't use the Accept header to
pick the format, instead we have explicitly specify the format with
/autocomplete.json.
This is clearer and better for caching anyway. Using the `Vary: Accept`
header reduces the cache hit rate, because the exact format of the
Accept header varies across browsers, which fragments the cache.
Whew.
[1] https://github.com/rails/rails/pull.36213
[2] https://community.cloudflare.com/t/cloudflare-cdn-cache-to-support-http-vary-header/160802
[3] https://support.cloudflare.com/hc/en-us/articles/115003206852
[4] https://www.smashingmagazine.com/2017/11/understanding-vary-header/
Fix session cookies being sent in publicly cached /autocomplete.json
responses. We can't set any cookies in a response that is being publicly
cached, otherwise they'll be visible to other users. If a user's session
cookies were to be cached, then it would allow their account to be stolen.
In reality, well-behaved caches like Cloudflare will simply refuse to
cache responses that contain cookies to avoid this scenario.
https://support.cloudflare.com/hc/en-us/articles/200172516-Understanding-Cloudflare-s-CDN:
BYPASS is returned when enabling Origin Cache-Control. Cloudflare also
sets BYPASS when your origin web server sends cookies in the response
header.
Fix an exception that could occur when typing "/" by itself in
autocomplete and a regular tag starting with "/" was returned. This
caused an exception in `r[:antecedent].length` because the tag's
antecedent was nil.
* Remove the `source` and `weight` html data attributes (no longer used).
* Make the `type` html data attribute properly indicate the completion
type. Valid types: `tag`, `tag-alias`, `tag-abbreviation`,
`tag-autocorrect`, `tag-other-name`.
Allowing typing Japanese tags in autocomplete. For example, typing 東方
in autocomplete will be completed to the touhou tag. Typing ぶくぶ will
complete to the bkub tag.
This works using wiki page and artist other names. Effectively, any name
listed as an other name in a wiki or artist page will be treated like an
alias for autocomplete purposes. This is limited to non-ASCII other names,
to prevent English other names from interfering with regular tag searches.
* Set Max-Age= flag instead of Expires= flag.
* Set Secure flag when using HTTPS.
* Extend default cookie lifetime from 1 year to 20 years.
* Remove "session" expiration option (unused).
* Remove max cookie size check.
The cookie size check was previously added in #2518 to deal with running
out of space due to tag scripts and blacklists. This should no longer
happen since we no longer use cookies for these things. Remove the
warning because it should never happen, we can't fix it if it does, and
the user probably won't know how to fix it either.
* Let Mods and Admins see the email addresses of users below their level.
* Let users see their own email address on their profile.
* Let users verify or edit their email address from their profile.
This is to make catching sockpuppets easier, and to make it easier for
users to fix their email.
Don't allow admins to bypass promotion restrictions by manually updating
user levels with a `PUT /users/:id` API call. Level changes have to go
through the /admin/users/:id/edit page.
Don't allow gifting Gold or Platinum upgrades to users above Platinum
level. Fixes an exploit where you could demote Builders and above by
gifting them an upgrade.
* Test that the user upgrade process integrates with Stripe correctly.
* Replace a deprecated `card` param with `source` in `Stripe::Charge.create`.
* Rescue Stripe::StripeError instead of Stripe::CardError so that we
handle failures outside of card failures, such as network errors.
New rules for user promotions:
* Moderators can no longer promote other users to moderator level. Only
Admins can promote users to Mod level. Mods can only promote up to Builder level.
* Admins can no longer promote other users to Admin level. Only Owners
can promote users to Admin. Admins can only promote up to Mod level.
* Admins can no longer demote themselves or other admins.
These rules are being changed to account for the new Owner user level.
Also change it so that when a user upgrades their account, the promotion
is done by DanbooruBot. This means that the inviter and the mod action
will show DanbooruBot as the promoter instead of the user themselves.
Add a new Owner user level for the site owner. Highly sensitive
operations like manually changing the passwords of other users will be
restricted to the site owner.
* Rename the `#negate` and `#and` methods that we monkey patch into
ActiveRecord::Relation. These methods are now defined in Rails 6.1, but
they shadow our methods and have slightly different behavior.
* Fix a call to `invert`. It no longer accepts an argument.
Upgrade the http-cookie gem to a personal fork containing a bugfix for a
http-cookie bug that is triggered by Rails 6.1.
The bug is that HTTP::Cookie objects raise an exception if they're
compared against non-cookie objects. This bug gets triggered when
the Nijie source strategy calls `Rails.cache.fetch` to cache the
Nijie login cookie. `Rails.cache.fetch` ends up calling
ActiveSupport::Cache::Store::Entry#dup_value!, which compares the cookie
with `true`, which triggers the exception.
The http-cookie gem hasn't been updated for 4 years, so we're stuck
patching the library ourselves.
Display autocorrected tags similar to aliases, with an arrow pointing at
the corrected tag, but with a dotted underline beneath the misspelled
tag to indicate that it's misspelled.
Tune autocorrect to produce fewer false positives. Before we used
trigram similarity. Now we use Levenshtein edit distance with a dynamic
typo threshold. Trigram similarity was able to correct large
transpositions (e.g. `miku_hatsune` -> `hatsune_miku`), but it was bad
at correcting small typos. Levenshtein is good at small typos, but can't
correct large transpositions.
Fix an exception that was thrown when trying to autocomplete saved
search labels (e.g. `search:all`) as an anonymous user. This was a
pre-existing bug.
The previous cache policy was that all autocomplete results were cached
for a fixed 7 days. The new policy is that if autocomplete returns more
than 10 results they're cached for 24 hours, otherwise if it returns
less than 10 results they're cached for 1 hour.
The rationale is that if autocomplete returns a lot of results, then the
top 10 results are relatively stable and unlikely to change, but if it
returns less than 10 results, then the results are unstable and can be
easily changed.
We also change it so that autocomplete calls can be cached publicly.
Public caching means that HTTP requests are cached by Cloudflare. This
will ideally reduce load on the server and reduce latency for end users.
This is only safe for calls that return the same results for all users
(i.e. the results don't depend on the current user), since the cache is
publicly shared by all users. Currently username, favgroup, and saved
search autocomplete results depend on the current user, so they can't be
publicly cached.
Reworks tag autocomplete to work the same way for all users. Previously
autocomplete for Builders worked differently than autocomplete for
regular users.
This is how it works now:
* If the search starts with a slash (/), then do a tag abbreviation
match. For example, `/evth` matches eyebrows_visible_through_hair.
* Otherwise if the search contains a wildcard (*), then just do a simple
wildcard search.
* Otherwise do a tag prefix match against tags and aliases. For example,
`black` matches all tags or aliases beginning with `black`.
* If the tag prefix match returns no results, then do a autocorrect match.
The differences for regular users:
* You can abbreviate tags with a slash (/).
The differences for Builders:
* Now tag abbreviations have to start with a slash (/).
* Autocorrect isn't performed unless a regular search returns no results.
* Results are always sorted by tag count. Before different types of
results (regular tag matches, alias matches, abbreviation matches,
and autocorrect matches) were all mixed together based on a tag
weighting scheme.
This refactors the autocomplete Javascript to use a single dedicated
/autocomplete.json endpoint instead of a bunch of separate endpoints.
This simplifies the autocomplete Javascript by making it so that instead
of calling a different endpoint for each type of query (for users, wiki
pages, pools, artists, etc), then having to parse the results of each
call to get the data we need, we can call a single endpoint that returns
exactly what we need.
This also means we don't have to parse searches clientside in order to
autocomplete metatags. Instead we can just pass the search term to the
server and let it parse the search, which is easy to do serverside.
Finally, this makes autocomplete easier to test, and it makes it easier
to add more sophisticated autocomplete behavior, since most of the logic
lives serverside.
Remove the "Repopulated 1 old tag" message. Show "Created 1 new tag"
instead. The distinction between creating a brand new tag and
repopulating an empty tag doesn't matter.
Bug: when aliasing a tag that implied another tag, it was possible for
the alias to fail. Moving the implication could fail because we checked
that the tag category of both tags in the implication was the same, but
we did this before the alias moved the category of the old tag to the
new tag.
When approving or rejecting a BUR, don't edit the OP forum post to add
an EDIT: line stating the request has been approved. Instead just let
the embedded BUR state who it was approved by, and post a reply saying
that the request has been approved.
Implications now have the following rules:
* The child tag must have at least 10 posts.
* The child tag must be at least 0.01% the size of the parent tag.
* The child tag can't make up more than 90% of the parent tag.
* These rules only apply to general tags.
Move the validation that the tags in an implication must have wiki pages
back into the TagImplication model. Use validation contexts to only run
the validation when the BUR is created, not when the BUR is approved.
Usage:
* `nuke touhou`
* `nuke pool:Disgustingly_Adorable`
Add a command for nuking tags. `nuke A` is a shortcut for `mass update A -> -A`.
This means it also works for pools.
Allow reversing an alias without having to remove the old alias first.
When aliasing A -> B, then if B -> A already exists it will
automatically be removed first.
Don't log mod actions when aliases, implications, or mass updates are
processed.
Originally aliases and implications were logged because they could be
approved outside of a BUR. Mass updates could also be performed by mods
without making a forum request. This is no longer the case.
They were also logged for debugging purposes. This is no longer needed.
This generated a lot of spam in the mod action logs when a large BUR was
approved.
Don't fragment cache the site news banner. Caching this trades a SQL
query for a Redis call, which is unlikely to make much performance
difference to page rendering but puts high traffic volume on Redis.
Bug: When validating a BUR, we didn't properly simulate running each
line of the BUR in order, which could cause validation to incorrectly
fail in multi-line BURs where some lines depended on previous lines.
This bug meant you couldn't reverse an alias in a single BUR. The old
alias wasn't removed before validating the new alias, so the BUR would
fail with an alias conflict.
This bug also meant that BURs containing duplicate aliases or
redundant implications weren't caught.
The fix is for BUR validation to actually simulate creating and removing
aliases in sequential order, just as they would be when the BUR is
approved. This is done by running the BUR in a transaction, then
rolling back the transaction at the end. This is hacky but it works.
When approving an implication, only retag posts that are missing the new
tag. Don't try to update posts that already have the tag. This makes
large implication requests faster to process when most of the posts
already have the implied tag.
Remove the error status from aliases and implications. Aliases and
implications normally shouldn't fail because they're validated
beforehand. If they do, just let the delayed job itself record the
failure.
Also disable the delayed job from retrying if the alias/implication
somehow fails.
Remove the pending status from tag aliases and implications.
Previously aliases would be created first in the pending state then
changed to active when the alias was later processed in a delayed job.
This meant that BURs weren't processed completely sequentially; first
all the aliases in a BUR would be created in one go, then later they
would be processed and set to active sequentially.
This was problematic in complex BURs that tried to reverse or swap
around aliases, since new pending aliases could be created before old
conflicting aliases were removed.
Remove the ability to skip secondary validations when creating a BUR.
The only skippable validation that still existed was the requirement
that both tags in an implication must have wiki pages. It's now
mandatory to write wiki pages for tags before you can request an
implication. This doesn't apply to empty tags.
Remove the `processing` state from aliases and implications. This state
was used to mark when an alias or implication had been approved but the
alias or implication was still being processed. Aliases in the
processing state were still considered active, so there was no
functional difference between the active state and the processing state.
This fixes a problem where it was possible for implications to get stuck
in the processing state. This happened when a BUR contained a duplicate
implication. Transitioning from the processing state to the active state
failed in this case because we used `update` instead of `update!`, which
meant validation errors were silently ignored.
Make the tag rename command also move any aliases or implications from
the old tag to the new tag. Previously only the create alias command
moved aliases and implications.
Bug: if a BUR contained a mass update followed by an alias, then the
alias would become active before the mass update, which could cause
the mass update to return incorrect results if both the alias and mass
update touched the same tags.
This happened because all aliases and implications in the BUR were set
to a queued state before the mass update was processed, but putting an
alias in the queued state effectively made it active.
The fix is to remove the queued state. This was only used anyway as a
debugging tool anyway to monitor the state of BURs as they were being
processed.
Bug: Verifying your account with a hotmail.com email address didn't mark
your account as verified. This was because the list of allowed email
verification domains only included normalized domains, but we were
looking up unnormalized domains.
Decrease the HTTP timeout to 0.5 seconds when fetching popular tags from
Reportbooru. Increase the length of time that popular tags are cached
from 1 minute to 1 hour. This is for the list of popular searches in the
front page sidebar.
When renaming a tag and the new tag has a qualifier, use the pipe trick
to hide the qualifier in wiki links. For example, renaming Fallout to
Fallout_(series) should change wiki links from [[Fallout]] to [[Fallout
(series)|]].
When a dmail is hard deleted, update the user's unread dmail count and
delete any associated mod reports.
Dmails aren't normally hard deleted, except when there's a large spam
wave that necessitates manual intervention.
Bug: if `hatsune_miku` is an empty general tag, then tagging a post with
`hatsune_miku_(cosplay)` would fail because `hatsune_miku` wasn't a
character tag. This could cause tagging posts to fail when aliasing a
character tag back to an old name that was used in the past, but was now
an empty general tag.
Fix: only treat the *_(cosplay) tag as being in conflict with the base
tag when the base tag is nonempty.
evazion
evazion