Stop maintaining pool category pseudo tags (pool:series, pool:collection)
in pool strings. They're no longer used and the changes to the
`Post#pools` method in dc4d2e54b caused issues with this.
Also allow Members to change the category of large pools again. This was
only restricted because maintaining these pseudotags forced us to update
every post in the pool whenever a pool's category was changed.
Stop using the pool_string field internally, but keep maintaining it
until we can drop it later.
* Stop using the pool_string for `pool:<name>` metatag searches.
* Stop using the pool_string in the `Post#pools` method. This is used to
get the list of pools on post show pages.
Revert optimization from a6163258b. Turns out that we have to resolve
aliases in fast_count, otherwise for aliased tags we'll return an empty
count.
Fixes#4156.
* Don't try to call `sadd` when a search returns no results (`sadd`
fails in this case).
* Add a timeout when populating the search.
* Don't offload the search to read replica. The main db is fine.
* Disable synchronous population of searches. This was too slow.
* Change the source index on posts from `(lower(source) gin_trgm_ops) WHERE source != ''`
to just `(source gin_trgm_ops)`. The WHERE clause prevented the index
from being used in source:<url> searches because we didn't specify
the `source != ''` clause in the search itself. Excluding blank
sources only saved a marginal amount of space anyway. This fixes
timeouts in source:<url> searches and in the bookmarklet (since we do
a source dupe check on the upload page too).
* Also switch from indexing `lower(name)` to `name` on pools and users.
We don't need to lowercase the column because GIN indexes can be used
with both LIKE and ILIKE queries.
Make /favorites redirect to a ordfav:<user> search instead of having a
separate view just for favorites. This duplicated a lot of code for no
good reason.
Rewrite the implementation of related tags to be simpler, faster, and
more accurate:
* The related tags are now calculated by taking a random sample of 1000
posts, finding the top 250 most frequent tags among those posts, then
ordering those tags by cosine similarity.
* Related tags can generally be calculated in 50-300ms at these sample
sizes. Very high sample sizes (25000+ posts) are still relatively fast
(1-3 seconds), but generally they don't improve accuracy much.
* Related tags are now cached in redis rather than in the tags table.
The related_tags column in the tags table is no longer used.
* Only the related tags in the search taglist are cached. The related
tags returned by the 'Related tags' button are not cached.
* The cache lifetime is a fixed 4 hours.
* The 'Related tags' button now works with metatags.
* The /related_tag page now works with metatags and multitag searches.
Fixes#4134, #4146.
* Drop support for `source:pixiv/artist-name` searches. This was a hack
that only worked on old pixiv urls that haven't been used for years.
* Replace the old SourcePattern(lower(source)) index with a trigram index.
Drop support for https://danbooru.donmai.us/cache/tags.json. This was a
nightly dump of the tags table that was originally added in #1012. It
was never documented and never really used except for by the DanbooruUp
extension.
Bug: sending dmails failed for members.
Cause: using lambdas with `rakismet_attrs` failed because unexpected
arguments are passed to the lambdas. Using procs works because the
arguments are ignored.
Also fix the tests to actually test akismet. We didn't catch this
because the tests mocked out the `spam?` call.
Certain parts of comment rendering triggered sql queries that we didn't
really need to do. Rework things to avoid this.
* Preload comment creators in order to display commenter names with link_to_user.
* Preload comment votes in order to display "undo vote" links. Only preload
votes for members since anonymous users can't vote and don't have "undo
vote" links.
* Rework various conditionals to do the filtering in Ruby so that we
avoid issuing any extra queries in sql.
* Avoid issuing any queries at all when the post doesn't have any
comments (when last_commented_at is blank).
Also fixes a bug where mod actions weren't logged on mass updates.
Creating the mod action silently failed because it was called when
CurrentUser wasn' set.
Changes:
* Drop Users.id_to_name.
* Don't cache Users.name_to_id.
* Replace calls to name_to_id with find_by_name when possible.
* Don't autodefine creator_name in belongs_to_creator.
* Don't autodefine updater_name in belongs_to_updater.
* Instead manually define creator_name / updater_name only on models that need
to return these fields in the api.
id_to_name was cached to reduce the impact of N+1 query patterns in
certain places, especially in api responses that return creator_name /
updater_name fields. But it still meant we were doing N calls to
memcache. Using `includes` to prefetch users avoids this N+1 pattern.
name_to_id had no need be cached, it was never used in any performance-
sensitive contexts.
Avoiding caching also avoids the need to keep these caches consistent.
Fix find_by_name and name_to_id to use normalize_name properly, so that
they ignore leading/trailing whitespace. This fixes various search forms
failing to return results when the username field contains trailing
whitespace (inserted by autocomplete).
* Fix tests to run the searches for real instead of mocking everything out.
* Fix SavedSearch.populate to only use the read only database in
production because in breaks things in tests. Specifically:
the posts get created in one db connection but searched for in
another, but the second transaction doesn't see the uncommitted posts
in the first transaction, so the search doesn't work.
* Don't allow adding tags with invalid names when they already exist in
the tags table.
* If an invalid tag is added, show an warning and ignore the tag instead
of failing with a hard error.
* Move the _(cosplay) tag validation into the tag name validator.
Bug: Searching for an aliased tag returns an empty page instead of
showing the results for the real tag.
Cause: The query parsing code runs the search string through
`.mb_chars.downcase` before calling `TagAlias.to_aliased`, so the input
to `to_aliased` is actually a ActiveSupport::Multibyte::Chars object.
This breaks the `aliases[name]` hash lookup because `name` is not a
plain string.
Fixup for c7bcce429.
