Expand the tag abbreviation system introduced in b0be8ae45 so that it
works in searches and when tagging posts, not just in autocomplete.
For example, you can tag a post with /evth and it will add the tag
eyebrows_visible_through_hair. You can search for /evth and it will
search for the tag eyebrows_visible_through_hair.
Some more examples:
* /ops is short for one-piece_swimsuit
* /hooe is short for hair_over_one_eye
* /saol is short for standing_on_one_leg
* /tlozbotw is short for the_legend_of_zelda:_breath_of_the_wild
If two tags have the same abbreviation, then the larger tag takes
precedence. For example, /be is short for blue_eyes, not brown_eyes,
because blue_eyes is the bigger tag.
If there is an existing shortcut alias that conflicts with the
abbreviation, then the alias take precedence. For example, /sh is short
for suzumiya_haruhi, not short_hair, because there's an old alias for
/sh -> suzumiya_haruhi.
Limit tag length to 170 chars. 170 chars was chosen because it's
longer than the longest active tag on Danbooru.
Tag length is limited because in some contexts we can't deal with
excessively long tags. Tag autocorrect for example uses the levenshtein
function in Postgres, which can't handle strings more than 255 chars long.
Disallow tags from starting with a '/' character. This is so that tag
abbreviations in autocomplete, which start with a '/', don't conflict
with regular tags.
Also disallow some other punctuation characters: `%{})]. Currently no
tags start with these characters. This is to reserve other special
characters in case we need them for other future syntax extensions.
Add the following search operators:
* /tags?search[post_count_eq]=42
* /tags?search[post_count_not_eq]=42
* /tags?search[post_count_gt]=42
* /tags?search[post_count_gteq]=42
* /tags?search[post_count_lt]=42
* /tags?search[post_count_lteq]=42
Works for all numeric attributes on all index actions.
This is the scenario:
* You type something in autocomplete, let's say 'touhou'.
* Autocomplete calls /autocomplete?search[query]=touhou&search[type]=tag_query
* The endpoint returns JSON, because the autocomplete call sets an
`Accept: application/json` header requesting JSON.
* Visit /autocomplete?search[query]=touhou&search[type]=tag_query in your browser.
* Notice that the cached JSON response is incorrectly returned, not an
HTML response like the browser requested.
The problem is that the response type is chosen based on the Accept
header, but the response didn't set the `Vary: Accept` header, so the
browser doesn't know the response type can vary and so it incorrectly
returns the cached response.
This issue is partially fixed by Rails 6.1 ([1]), which properly sets the
`Vary: Accept` header when the response depends on the Accept header.
However, the next issue is that Cloudflare doesn't respect the Vary
header at all ([2], [3]). Therefore we can't use the Accept header to
pick the format, instead we have explicitly specify the format with
/autocomplete.json.
This is clearer and better for caching anyway. Using the `Vary: Accept`
header reduces the cache hit rate, because the exact format of the
Accept header varies across browsers, which fragments the cache.
Whew.
[1] https://github.com/rails/rails/pull.36213
[2] https://community.cloudflare.com/t/cloudflare-cdn-cache-to-support-http-vary-header/160802
[3] https://support.cloudflare.com/hc/en-us/articles/115003206852
[4] https://www.smashingmagazine.com/2017/11/understanding-vary-header/
Fix session cookies being sent in publicly cached /autocomplete.json
responses. We can't set any cookies in a response that is being publicly
cached, otherwise they'll be visible to other users. If a user's session
cookies were to be cached, then it would allow their account to be stolen.
In reality, well-behaved caches like Cloudflare will simply refuse to
cache responses that contain cookies to avoid this scenario.
https://support.cloudflare.com/hc/en-us/articles/200172516-Understanding-Cloudflare-s-CDN:
BYPASS is returned when enabling Origin Cache-Control. Cloudflare also
sets BYPASS when your origin web server sends cookies in the response
header.
Fix an exception that could occur when typing "/" by itself in
autocomplete and a regular tag starting with "/" was returned. This
caused an exception in `r[:antecedent].length` because the tag's
antecedent was nil.
* Remove the `source` and `weight` html data attributes (no longer used).
* Make the `type` html data attribute properly indicate the completion
type. Valid types: `tag`, `tag-alias`, `tag-abbreviation`,
`tag-autocorrect`, `tag-other-name`.
Allowing typing Japanese tags in autocomplete. For example, typing 東方
in autocomplete will be completed to the touhou tag. Typing ぶくぶ will
complete to the bkub tag.
This works using wiki page and artist other names. Effectively, any name
listed as an other name in a wiki or artist page will be treated like an
alias for autocomplete purposes. This is limited to non-ASCII other names,
to prevent English other names from interfering with regular tag searches.
* Set Max-Age= flag instead of Expires= flag.
* Set Secure flag when using HTTPS.
* Extend default cookie lifetime from 1 year to 20 years.
* Remove "session" expiration option (unused).
* Remove max cookie size check.
The cookie size check was previously added in #2518 to deal with running
out of space due to tag scripts and blacklists. This should no longer
happen since we no longer use cookies for these things. Remove the
warning because it should never happen, we can't fix it if it does, and
the user probably won't know how to fix it either.
* Let Mods and Admins see the email addresses of users below their level.
* Let users see their own email address on their profile.
* Let users verify or edit their email address from their profile.
This is to make catching sockpuppets easier, and to make it easier for
users to fix their email.
Don't allow admins to bypass promotion restrictions by manually updating
user levels with a `PUT /users/:id` API call. Level changes have to go
through the /admin/users/:id/edit page.
Don't allow gifting Gold or Platinum upgrades to users above Platinum
level. Fixes an exploit where you could demote Builders and above by
gifting them an upgrade.
* Test that the user upgrade process integrates with Stripe correctly.
* Replace a deprecated `card` param with `source` in `Stripe::Charge.create`.
* Rescue Stripe::StripeError instead of Stripe::CardError so that we
handle failures outside of card failures, such as network errors.
New rules for user promotions:
* Moderators can no longer promote other users to moderator level. Only
Admins can promote users to Mod level. Mods can only promote up to Builder level.
* Admins can no longer promote other users to Admin level. Only Owners
can promote users to Admin. Admins can only promote up to Mod level.
* Admins can no longer demote themselves or other admins.
These rules are being changed to account for the new Owner user level.
Also change it so that when a user upgrades their account, the promotion
is done by DanbooruBot. This means that the inviter and the mod action
will show DanbooruBot as the promoter instead of the user themselves.
Add a new Owner user level for the site owner. Highly sensitive
operations like manually changing the passwords of other users will be
restricted to the site owner.
* Rename the `#negate` and `#and` methods that we monkey patch into
ActiveRecord::Relation. These methods are now defined in Rails 6.1, but
they shadow our methods and have slightly different behavior.
* Fix a call to `invert`. It no longer accepts an argument.
Upgrade the http-cookie gem to a personal fork containing a bugfix for a
http-cookie bug that is triggered by Rails 6.1.
The bug is that HTTP::Cookie objects raise an exception if they're
compared against non-cookie objects. This bug gets triggered when
the Nijie source strategy calls `Rails.cache.fetch` to cache the
Nijie login cookie. `Rails.cache.fetch` ends up calling
ActiveSupport::Cache::Store::Entry#dup_value!, which compares the cookie
with `true`, which triggers the exception.
The http-cookie gem hasn't been updated for 4 years, so we're stuck
patching the library ourselves.
Display autocorrected tags similar to aliases, with an arrow pointing at
the corrected tag, but with a dotted underline beneath the misspelled
tag to indicate that it's misspelled.
Tune autocorrect to produce fewer false positives. Before we used
trigram similarity. Now we use Levenshtein edit distance with a dynamic
typo threshold. Trigram similarity was able to correct large
transpositions (e.g. `miku_hatsune` -> `hatsune_miku`), but it was bad
at correcting small typos. Levenshtein is good at small typos, but can't
correct large transpositions.
Fix an exception that was thrown when trying to autocomplete saved
search labels (e.g. `search:all`) as an anonymous user. This was a
pre-existing bug.
The previous cache policy was that all autocomplete results were cached
for a fixed 7 days. The new policy is that if autocomplete returns more
than 10 results they're cached for 24 hours, otherwise if it returns
less than 10 results they're cached for 1 hour.
The rationale is that if autocomplete returns a lot of results, then the
top 10 results are relatively stable and unlikely to change, but if it
returns less than 10 results, then the results are unstable and can be
easily changed.
We also change it so that autocomplete calls can be cached publicly.
Public caching means that HTTP requests are cached by Cloudflare. This
will ideally reduce load on the server and reduce latency for end users.
This is only safe for calls that return the same results for all users
(i.e. the results don't depend on the current user), since the cache is
publicly shared by all users. Currently username, favgroup, and saved
search autocomplete results depend on the current user, so they can't be
publicly cached.
Reworks tag autocomplete to work the same way for all users. Previously
autocomplete for Builders worked differently than autocomplete for
regular users.
This is how it works now:
* If the search starts with a slash (/), then do a tag abbreviation
match. For example, `/evth` matches eyebrows_visible_through_hair.
* Otherwise if the search contains a wildcard (*), then just do a simple
wildcard search.
* Otherwise do a tag prefix match against tags and aliases. For example,
`black` matches all tags or aliases beginning with `black`.
* If the tag prefix match returns no results, then do a autocorrect match.
The differences for regular users:
* You can abbreviate tags with a slash (/).
The differences for Builders:
* Now tag abbreviations have to start with a slash (/).
* Autocorrect isn't performed unless a regular search returns no results.
* Results are always sorted by tag count. Before different types of
results (regular tag matches, alias matches, abbreviation matches,
and autocorrect matches) were all mixed together based on a tag
weighting scheme.
This refactors the autocomplete Javascript to use a single dedicated
/autocomplete.json endpoint instead of a bunch of separate endpoints.
This simplifies the autocomplete Javascript by making it so that instead
of calling a different endpoint for each type of query (for users, wiki
pages, pools, artists, etc), then having to parse the results of each
call to get the data we need, we can call a single endpoint that returns
exactly what we need.
This also means we don't have to parse searches clientside in order to
autocomplete metatags. Instead we can just pass the search term to the
server and let it parse the search, which is easy to do serverside.
Finally, this makes autocomplete easier to test, and it makes it easier
to add more sophisticated autocomplete behavior, since most of the logic
lives serverside.
Remove the "Repopulated 1 old tag" message. Show "Created 1 new tag"
instead. The distinction between creating a brand new tag and
repopulating an empty tag doesn't matter.
Bug: when aliasing a tag that implied another tag, it was possible for
the alias to fail. Moving the implication could fail because we checked
that the tag category of both tags in the implication was the same, but
we did this before the alias moved the category of the old tag to the
new tag.
