Expand the tag abbreviation system introduced in b0be8ae45 so that it
works in searches and when tagging posts, not just in autocomplete.
For example, you can tag a post with /evth and it will add the tag
eyebrows_visible_through_hair. You can search for /evth and it will
search for the tag eyebrows_visible_through_hair.
Some more examples:
* /ops is short for one-piece_swimsuit
* /hooe is short for hair_over_one_eye
* /saol is short for standing_on_one_leg
* /tlozbotw is short for the_legend_of_zelda:_breath_of_the_wild
If two tags have the same abbreviation, then the larger tag takes
precedence. For example, /be is short for blue_eyes, not brown_eyes,
because blue_eyes is the bigger tag.
If there is an existing shortcut alias that conflicts with the
abbreviation, then the alias take precedence. For example, /sh is short
for suzumiya_haruhi, not short_hair, because there's an old alias for
/sh -> suzumiya_haruhi.
Limit tag length to 170 chars. 170 chars was chosen because it's
longer than the longest active tag on Danbooru.
Tag length is limited because in some contexts we can't deal with
excessively long tags. Tag autocorrect for example uses the levenshtein
function in Postgres, which can't handle strings more than 255 chars long.
Disallow tags from starting with a '/' character. This is so that tag
abbreviations in autocomplete, which start with a '/', don't conflict
with regular tags.
Also disallow some other punctuation characters: `%{})]. Currently no
tags start with these characters. This is to reserve other special
characters in case we need them for other future syntax extensions.
Add the following search operators:
* /tags?search[post_count_eq]=42
* /tags?search[post_count_not_eq]=42
* /tags?search[post_count_gt]=42
* /tags?search[post_count_gteq]=42
* /tags?search[post_count_lt]=42
* /tags?search[post_count_lteq]=42
Works for all numeric attributes on all index actions.
Fix an exception that could occur when typing "/" by itself in
autocomplete and a regular tag starting with "/" was returned. This
caused an exception in `r[:antecedent].length` because the tag's
antecedent was nil.
* Remove the `source` and `weight` html data attributes (no longer used).
* Make the `type` html data attribute properly indicate the completion
type. Valid types: `tag`, `tag-alias`, `tag-abbreviation`,
`tag-autocorrect`, `tag-other-name`.
Allowing typing Japanese tags in autocomplete. For example, typing 東方
in autocomplete will be completed to the touhou tag. Typing ぶくぶ will
complete to the bkub tag.
This works using wiki page and artist other names. Effectively, any name
listed as an other name in a wiki or artist page will be treated like an
alias for autocomplete purposes. This is limited to non-ASCII other names,
to prevent English other names from interfering with regular tag searches.
Don't allow gifting Gold or Platinum upgrades to users above Platinum
level. Fixes an exploit where you could demote Builders and above by
gifting them an upgrade.
New rules for user promotions:
* Moderators can no longer promote other users to moderator level. Only
Admins can promote users to Mod level. Mods can only promote up to Builder level.
* Admins can no longer promote other users to Admin level. Only Owners
can promote users to Admin. Admins can only promote up to Mod level.
* Admins can no longer demote themselves or other admins.
These rules are being changed to account for the new Owner user level.
Also change it so that when a user upgrades their account, the promotion
is done by DanbooruBot. This means that the inviter and the mod action
will show DanbooruBot as the promoter instead of the user themselves.
Add a new Owner user level for the site owner. Highly sensitive
operations like manually changing the passwords of other users will be
restricted to the site owner.
* Rename the `#negate` and `#and` methods that we monkey patch into
ActiveRecord::Relation. These methods are now defined in Rails 6.1, but
they shadow our methods and have slightly different behavior.
* Fix a call to `invert`. It no longer accepts an argument.
Display autocorrected tags similar to aliases, with an arrow pointing at
the corrected tag, but with a dotted underline beneath the misspelled
tag to indicate that it's misspelled.
Tune autocorrect to produce fewer false positives. Before we used
trigram similarity. Now we use Levenshtein edit distance with a dynamic
typo threshold. Trigram similarity was able to correct large
transpositions (e.g. `miku_hatsune` -> `hatsune_miku`), but it was bad
at correcting small typos. Levenshtein is good at small typos, but can't
correct large transpositions.
Fix an exception that was thrown when trying to autocomplete saved
search labels (e.g. `search:all`) as an anonymous user. This was a
pre-existing bug.
The previous cache policy was that all autocomplete results were cached
for a fixed 7 days. The new policy is that if autocomplete returns more
than 10 results they're cached for 24 hours, otherwise if it returns
less than 10 results they're cached for 1 hour.
The rationale is that if autocomplete returns a lot of results, then the
top 10 results are relatively stable and unlikely to change, but if it
returns less than 10 results, then the results are unstable and can be
easily changed.
We also change it so that autocomplete calls can be cached publicly.
Public caching means that HTTP requests are cached by Cloudflare. This
will ideally reduce load on the server and reduce latency for end users.
This is only safe for calls that return the same results for all users
(i.e. the results don't depend on the current user), since the cache is
publicly shared by all users. Currently username, favgroup, and saved
search autocomplete results depend on the current user, so they can't be
publicly cached.
Reworks tag autocomplete to work the same way for all users. Previously
autocomplete for Builders worked differently than autocomplete for
regular users.
This is how it works now:
* If the search starts with a slash (/), then do a tag abbreviation
match. For example, `/evth` matches eyebrows_visible_through_hair.
* Otherwise if the search contains a wildcard (*), then just do a simple
wildcard search.
* Otherwise do a tag prefix match against tags and aliases. For example,
`black` matches all tags or aliases beginning with `black`.
* If the tag prefix match returns no results, then do a autocorrect match.
The differences for regular users:
* You can abbreviate tags with a slash (/).
The differences for Builders:
* Now tag abbreviations have to start with a slash (/).
* Autocorrect isn't performed unless a regular search returns no results.
* Results are always sorted by tag count. Before different types of
results (regular tag matches, alias matches, abbreviation matches,
and autocorrect matches) were all mixed together based on a tag
weighting scheme.
This refactors the autocomplete Javascript to use a single dedicated
/autocomplete.json endpoint instead of a bunch of separate endpoints.
This simplifies the autocomplete Javascript by making it so that instead
of calling a different endpoint for each type of query (for users, wiki
pages, pools, artists, etc), then having to parse the results of each
call to get the data we need, we can call a single endpoint that returns
exactly what we need.
This also means we don't have to parse searches clientside in order to
autocomplete metatags. Instead we can just pass the search term to the
server and let it parse the search, which is easy to do serverside.
Finally, this makes autocomplete easier to test, and it makes it easier
to add more sophisticated autocomplete behavior, since most of the logic
lives serverside.
Bug: when aliasing a tag that implied another tag, it was possible for
the alias to fail. Moving the implication could fail because we checked
that the tag category of both tags in the implication was the same, but
we did this before the alias moved the category of the old tag to the
new tag.
When approving or rejecting a BUR, don't edit the OP forum post to add
an EDIT: line stating the request has been approved. Instead just let
the embedded BUR state who it was approved by, and post a reply saying
that the request has been approved.
Move the validation that the tags in an implication must have wiki pages
back into the TagImplication model. Use validation contexts to only run
the validation when the BUR is created, not when the BUR is approved.
Usage:
* `nuke touhou`
* `nuke pool:Disgustingly_Adorable`
Add a command for nuking tags. `nuke A` is a shortcut for `mass update A -> -A`.
This means it also works for pools.
Bug: When validating a BUR, we didn't properly simulate running each
line of the BUR in order, which could cause validation to incorrectly
fail in multi-line BURs where some lines depended on previous lines.
This bug meant you couldn't reverse an alias in a single BUR. The old
alias wasn't removed before validating the new alias, so the BUR would
fail with an alias conflict.
This bug also meant that BURs containing duplicate aliases or
redundant implications weren't caught.
The fix is for BUR validation to actually simulate creating and removing
aliases in sequential order, just as they would be when the BUR is
approved. This is done by running the BUR in a transaction, then
rolling back the transaction at the end. This is hacky but it works.
Remove the error status from aliases and implications. Aliases and
implications normally shouldn't fail because they're validated
beforehand. If they do, just let the delayed job itself record the
failure.
Also disable the delayed job from retrying if the alias/implication
somehow fails.
Remove the pending status from tag aliases and implications.
Previously aliases would be created first in the pending state then
changed to active when the alias was later processed in a delayed job.
This meant that BURs weren't processed completely sequentially; first
all the aliases in a BUR would be created in one go, then later they
would be processed and set to active sequentially.
This was problematic in complex BURs that tried to reverse or swap
around aliases, since new pending aliases could be created before old
conflicting aliases were removed.
Remove the ability to skip secondary validations when creating a BUR.
The only skippable validation that still existed was the requirement
that both tags in an implication must have wiki pages. It's now
mandatory to write wiki pages for tags before you can request an
implication. This doesn't apply to empty tags.
Remove the `processing` state from aliases and implications. This state
was used to mark when an alias or implication had been approved but the
alias or implication was still being processed. Aliases in the
processing state were still considered active, so there was no
functional difference between the active state and the processing state.
This fixes a problem where it was possible for implications to get stuck
in the processing state. This happened when a BUR contained a duplicate
implication. Transitioning from the processing state to the active state
failed in this case because we used `update` instead of `update!`, which
meant validation errors were silently ignored.
Make the tag rename command also move any aliases or implications from
the old tag to the new tag. Previously only the create alias command
moved aliases and implications.
Bug: if a BUR contained a mass update followed by an alias, then the
alias would become active before the mass update, which could cause
the mass update to return incorrect results if both the alias and mass
update touched the same tags.
This happened because all aliases and implications in the BUR were set
to a queued state before the mass update was processed, but putting an
alias in the queued state effectively made it active.
The fix is to remove the queued state. This was only used anyway as a
debugging tool anyway to monitor the state of BURs as they were being
processed.
Decrease the HTTP timeout to 0.5 seconds when fetching popular tags from
Reportbooru. Increase the length of time that popular tags are cached
from 1 minute to 1 hour. This is for the list of popular searches in the
front page sidebar.
When renaming a tag and the new tag has a qualifier, use the pipe trick
to hide the qualifier in wiki links. For example, renaming Fallout to
Fallout_(series) should change wiki links from [[Fallout]] to [[Fallout
(series)|]].
Bug: if `hatsune_miku` is an empty general tag, then tagging a post with
`hatsune_miku_(cosplay)` would fail because `hatsune_miku` wasn't a
character tag. This could cause tagging posts to fail when aliasing a
character tag back to an old name that was used in the past, but was now
an empty general tag.
Fix: only treat the *_(cosplay) tag as being in conflict with the base
tag when the base tag is nonempty.
