Expand the tag abbreviation system introduced in b0be8ae45 so that it
works in searches and when tagging posts, not just in autocomplete.
For example, you can tag a post with /evth and it will add the tag
eyebrows_visible_through_hair. You can search for /evth and it will
search for the tag eyebrows_visible_through_hair.
Some more examples:
* /ops is short for one-piece_swimsuit
* /hooe is short for hair_over_one_eye
* /saol is short for standing_on_one_leg
* /tlozbotw is short for the_legend_of_zelda:_breath_of_the_wild
If two tags have the same abbreviation, then the larger tag takes
precedence. For example, /be is short for blue_eyes, not brown_eyes,
because blue_eyes is the bigger tag.
If there is an existing shortcut alias that conflicts with the
abbreviation, then the alias take precedence. For example, /sh is short
for suzumiya_haruhi, not short_hair, because there's an old alias for
/sh -> suzumiya_haruhi.
Allowing typing Japanese tags in autocomplete. For example, typing 東方
in autocomplete will be completed to the touhou tag. Typing ぶくぶ will
complete to the bkub tag.
This works using wiki page and artist other names. Effectively, any name
listed as an other name in a wiki or artist page will be treated like an
alias for autocomplete purposes. This is limited to non-ASCII other names,
to prevent English other names from interfering with regular tag searches.
New rules for user promotions:
* Moderators can no longer promote other users to moderator level. Only
Admins can promote users to Mod level. Mods can only promote up to Builder level.
* Admins can no longer promote other users to Admin level. Only Owners
can promote users to Admin. Admins can only promote up to Mod level.
* Admins can no longer demote themselves or other admins.
These rules are being changed to account for the new Owner user level.
Also change it so that when a user upgrades their account, the promotion
is done by DanbooruBot. This means that the inviter and the mod action
will show DanbooruBot as the promoter instead of the user themselves.
Add a new Owner user level for the site owner. Highly sensitive
operations like manually changing the passwords of other users will be
restricted to the site owner.
Tune autocorrect to produce fewer false positives. Before we used
trigram similarity. Now we use Levenshtein edit distance with a dynamic
typo threshold. Trigram similarity was able to correct large
transpositions (e.g. `miku_hatsune` -> `hatsune_miku`), but it was bad
at correcting small typos. Levenshtein is good at small typos, but can't
correct large transpositions.
Fix an exception that was thrown when trying to autocomplete saved
search labels (e.g. `search:all`) as an anonymous user. This was a
pre-existing bug.
Reworks tag autocomplete to work the same way for all users. Previously
autocomplete for Builders worked differently than autocomplete for
regular users.
This is how it works now:
* If the search starts with a slash (/), then do a tag abbreviation
match. For example, `/evth` matches eyebrows_visible_through_hair.
* Otherwise if the search contains a wildcard (*), then just do a simple
wildcard search.
* Otherwise do a tag prefix match against tags and aliases. For example,
`black` matches all tags or aliases beginning with `black`.
* If the tag prefix match returns no results, then do a autocorrect match.
The differences for regular users:
* You can abbreviate tags with a slash (/).
The differences for Builders:
* Now tag abbreviations have to start with a slash (/).
* Autocorrect isn't performed unless a regular search returns no results.
* Results are always sorted by tag count. Before different types of
results (regular tag matches, alias matches, abbreviation matches,
and autocorrect matches) were all mixed together based on a tag
weighting scheme.
This refactors the autocomplete Javascript to use a single dedicated
/autocomplete.json endpoint instead of a bunch of separate endpoints.
This simplifies the autocomplete Javascript by making it so that instead
of calling a different endpoint for each type of query (for users, wiki
pages, pools, artists, etc), then having to parse the results of each
call to get the data we need, we can call a single endpoint that returns
exactly what we need.
This also means we don't have to parse searches clientside in order to
autocomplete metatags. Instead we can just pass the search term to the
server and let it parse the search, which is easy to do serverside.
Finally, this makes autocomplete easier to test, and it makes it easier
to add more sophisticated autocomplete behavior, since most of the logic
lives serverside.
Remove the "Repopulated 1 old tag" message. Show "Created 1 new tag"
instead. The distinction between creating a brand new tag and
repopulating an empty tag doesn't matter.
Bug: when aliasing a tag that implied another tag, it was possible for
the alias to fail. Moving the implication could fail because we checked
that the tag category of both tags in the implication was the same, but
we did this before the alias moved the category of the old tag to the
new tag.
When approving or rejecting a BUR, don't edit the OP forum post to add
an EDIT: line stating the request has been approved. Instead just let
the embedded BUR state who it was approved by, and post a reply saying
that the request has been approved.
Implications now have the following rules:
* The child tag must have at least 10 posts.
* The child tag must be at least 0.01% the size of the parent tag.
* The child tag can't make up more than 90% of the parent tag.
* These rules only apply to general tags.
Move the validation that the tags in an implication must have wiki pages
back into the TagImplication model. Use validation contexts to only run
the validation when the BUR is created, not when the BUR is approved.
Allow reversing an alias without having to remove the old alias first.
When aliasing A -> B, then if B -> A already exists it will
automatically be removed first.
Don't log mod actions when aliases, implications, or mass updates are
processed.
Originally aliases and implications were logged because they could be
approved outside of a BUR. Mass updates could also be performed by mods
without making a forum request. This is no longer the case.
They were also logged for debugging purposes. This is no longer needed.
This generated a lot of spam in the mod action logs when a large BUR was
approved.
When approving an implication, only retag posts that are missing the new
tag. Don't try to update posts that already have the tag. This makes
large implication requests faster to process when most of the posts
already have the implied tag.
Remove the error status from aliases and implications. Aliases and
implications normally shouldn't fail because they're validated
beforehand. If they do, just let the delayed job itself record the
failure.
Also disable the delayed job from retrying if the alias/implication
somehow fails.
Remove the pending status from tag aliases and implications.
Previously aliases would be created first in the pending state then
changed to active when the alias was later processed in a delayed job.
This meant that BURs weren't processed completely sequentially; first
all the aliases in a BUR would be created in one go, then later they
would be processed and set to active sequentially.
This was problematic in complex BURs that tried to reverse or swap
around aliases, since new pending aliases could be created before old
conflicting aliases were removed.
Remove the ability to skip secondary validations when creating a BUR.
The only skippable validation that still existed was the requirement
that both tags in an implication must have wiki pages. It's now
mandatory to write wiki pages for tags before you can request an
implication. This doesn't apply to empty tags.
Remove the `processing` state from aliases and implications. This state
was used to mark when an alias or implication had been approved but the
alias or implication was still being processed. Aliases in the
processing state were still considered active, so there was no
functional difference between the active state and the processing state.
This fixes a problem where it was possible for implications to get stuck
in the processing state. This happened when a BUR contained a duplicate
implication. Transitioning from the processing state to the active state
failed in this case because we used `update` instead of `update!`, which
meant validation errors were silently ignored.
Make the tag rename command also move any aliases or implications from
the old tag to the new tag. Previously only the create alias command
moved aliases and implications.
Bug: if a BUR contained a mass update followed by an alias, then the
alias would become active before the mass update, which could cause
the mass update to return incorrect results if both the alias and mass
update touched the same tags.
This happened because all aliases and implications in the BUR were set
to a queued state before the mass update was processed, but putting an
alias in the queued state effectively made it active.
The fix is to remove the queued state. This was only used anyway as a
debugging tool anyway to monitor the state of BURs as they were being
processed.
Bug: Verifying your account with a hotmail.com email address didn't mark
your account as verified. This was because the list of allowed email
verification domains only included normalized domains, but we were
looking up unnormalized domains.
When a dmail is hard deleted, update the user's unread dmail count and
delete any associated mod reports.
Dmails aren't normally hard deleted, except when there's a large spam
wave that necessitates manual intervention.
When aliasing A to B, update any wikis linking to [[A]] to link to [[B]]
instead.
This is a best-effort process based on rough heuristics. There are a few
known problems:
* We don't always know how to capitalize the new tag. We try to mimic
the capitalization of the old tag, such that if the old tag was
capitalized (because it was at the beginning of a sentence), or if
every word in the old link was capitalized (because it's a proper
noun), then the new link will be capitalized in the same way. This can
handle simple general tags and character tags, but will fail for
copyright tags with mixed capitalization. For example, we don't know
that [[jojo_no_kimyou_na_bouken]] should be capitalized as [[JoJo no
Kimyou na Bouken]]. If we don't know how to capitalize the new tag, we
leave the old tag as-is so it can manually be fixed.
* Some aliases might require changing how a tag is pluralized. If we
changed [[rat]] to [[mouse]], then we should change `[[rat]]s` to
[[mice]]. We don't try to deal with this.
* In general, some changes might require entire sentences to be
rewritten to keep the grammar correct. Changing something like
[[skirt lift]] to [[lifting skirt]] could break the grammar of the
sentence. We don't try to deal with this.
* Factor out the code for moving tags from tag aliases to a separate
TagMover class.
* When aliasing two tags that have conflicting wikis, merge the old wiki
into the new one instead of failing with an error. Merge the other names
fields, replace the old wiki body with a message linking to the new
wiki, and mark the old wiki as deleted.
* When aliasing two tags that have conflicting artist entries, merge the
old artist into the new one instead of silently ignore the conflict.
Merge the group name, other names, and urls fields, and mark the old
artist as deleted.
* When two tags have conflicting wikis or artist entries, but the old
wiki or artist entry is deleted, then just ignore the old wiki or
artist and don't try to merge it.
* Fix it so that when saved searches are rewritten, we rewrite negated
searches too.
Only downcase tags in aliases, implications, and category change
commands. Don't downcase mass update commands. Mass updates are
potentially case sensitive (for example: `mass update source:imageboard -> source:Imageboard`).
* Don't raise exceptions when a BUR is invalid. Instead, use Rails
validations to return errors. Fixes invalid BURs potentially raising
exceptions in views. Also makes it so that each error in a BUR is
reported, not just the first one.
* Revalidate the BUR whenever the script is edited, not just when the
BUR is created. Ensures the BUR can't be broken by editing. Fixes a bug
where forum threads could be broken by someone editing a BUR and
breaking the syntax, thereby causing the BUR to raise an unparseable
script error when the forum thread was viewed.
* Validate that removed aliases and implication actually exist.
* Validate that the tag actually exists when changing a tag's category.
* Combine bulk update request processor unit tests with main bulk update
request unit tests.
Move html_data_attributes definitions from models to policies. Which
attributes are permitted as data-* attributes is a view level concern
and should be defined on the policy level, not the model level. Models
should be agnostic about how they're used in views.
