Revert optimization from a6163258b. Turns out that we have to resolve
aliases in fast_count, otherwise for aliased tags we'll return an empty
count.
Fixes#4156.
Don't try delete files belonging to completed uploads during pruning. If
an uploaded post was replaced shortly after upload, then the original
file could get deleted during pruning since it's no longer in use, but
this isn't supposed to happen until the replacement grace period (30
days) is over.
* Don't try to call `sadd` when a search returns no results (`sadd`
fails in this case).
* Add a timeout when populating the search.
* Don't offload the search to read replica. The main db is fine.
* Disable synchronous population of searches. This was too slow.
* Change the source index on posts from `(lower(source) gin_trgm_ops) WHERE source != ''`
to just `(source gin_trgm_ops)`. The WHERE clause prevented the index
from being used in source:<url> searches because we didn't specify
the `source != ''` clause in the search itself. Excluding blank
sources only saved a marginal amount of space anyway. This fixes
timeouts in source:<url> searches and in the bookmarklet (since we do
a source dupe check on the upload page too).
* Also switch from indexing `lower(name)` to `name` on pools and users.
We don't need to lowercase the column because GIN indexes can be used
with both LIKE and ILIKE queries.
DEPRECATION WARNING: Dangerous query method (method whose arguments
are used as raw SQL) called with non-attribute argument(s): "id =
52664 desc". Non-attribute arguments will be disallowed in Rails
6.1. This method should not be called with user-provided values,
such as request parameters or model attributes. Known-safe values
can be passed by wrapping them in Arel.sql().
Comments have three states: visible, hidden, and invisible. Visible
comments are always shown. Hidden comments are not shown until the user
clicks 'Show all comments'. Invisible comments are never shown to the
user. Deleted comments are treated as hidden for moderators and
invisible for normal users. Thresholded comments are treated as hidden
for all users.
Rewrite the implementation of related tags to be simpler, faster, and
more accurate:
* The related tags are now calculated by taking a random sample of 1000
posts, finding the top 250 most frequent tags among those posts, then
ordering those tags by cosine similarity.
* Related tags can generally be calculated in 50-300ms at these sample
sizes. Very high sample sizes (25000+ posts) are still relatively fast
(1-3 seconds), but generally they don't improve accuracy much.
* Related tags are now cached in redis rather than in the tags table.
The related_tags column in the tags table is no longer used.
* Only the related tags in the search taglist are cached. The related
tags returned by the 'Related tags' button are not cached.
* The cache lifetime is a fixed 4 hours.
* The 'Related tags' button now works with metatags.
* The /related_tag page now works with metatags and multitag searches.
Fixes#4134, #4146.
* Drop support for `source:pixiv/artist-name` searches. This was a hack
that only worked on old pixiv urls that haven't been used for years.
* Replace the old SourcePattern(lower(source)) index with a trigram index.
Previously if a pool had >100 posts then the 'Order' link wouldn't
appear in the navbar, but it was still possible to visit the pool order
page directly. If a user did so, only the first 100 posts in the pool
would be shown in the drag-and-drop widget. If they tried to reorder the
pool anyway, then everything beyond the first 100 posts would be
silently removed from the pool.
Now we always show the 'Order' link, but we disable the drag-and-drop
reordering widget when the pool has >100 posts to prevent posts from
being silently removed.
Fixes a conflict with `Enumerable#excluding` in Rails 6.
Rename Array#without and Enumerable#without to Array#excluding and
Enumerable#excluding. Old method names are retained as aliases.
Bug: sending dmails failed for members.
Cause: using lambdas with `rakismet_attrs` failed because unexpected
arguments are passed to the lambdas. Using procs works because the
arguments are ignored.
Also fix the tests to actually test akismet. We didn't catch this
because the tests mocked out the `spam?` call.
Certain parts of comment rendering triggered sql queries that we didn't
really need to do. Rework things to avoid this.
* Preload comment creators in order to display commenter names with link_to_user.
* Preload comment votes in order to display "undo vote" links. Only preload
votes for members since anonymous users can't vote and don't have "undo
vote" links.
* Rework various conditionals to do the filtering in Ruby so that we
avoid issuing any extra queries in sql.
* Avoid issuing any queries at all when the post doesn't have any
comments (when last_commented_at is blank).
Changes to the /wiki_page_versions global listing:
* Add "diff" links that show you what changed in the given edit.
* Add "?" links that take you to the current version of the wiki.
* Add "»" links next to wiki page titles that take you to the wiki's full edit history.
* Add "»" links next to usernames that take you to the user's full edit history.
* Add a "Status" column that shows whether the wiki page was created,
deleted, undeleted, or renamed.
* Link to /wiki_page_versions in sidebar, not /wiki_pages?order=time.
Changes:
* Drop Users.id_to_name.
* Don't cache Users.name_to_id.
* Replace calls to name_to_id with find_by_name when possible.
* Don't autodefine creator_name in belongs_to_creator.
* Don't autodefine updater_name in belongs_to_updater.
* Instead manually define creator_name / updater_name only on models that need
to return these fields in the api.
id_to_name was cached to reduce the impact of N+1 query patterns in
certain places, especially in api responses that return creator_name /
updater_name fields. But it still meant we were doing N calls to
memcache. Using `includes` to prefetch users avoids this N+1 pattern.
name_to_id had no need be cached, it was never used in any performance-
sensitive contexts.
Avoiding caching also avoids the need to keep these caches consistent.
Fix find_by_name and name_to_id to use normalize_name properly, so that
they ignore leading/trailing whitespace. This fixes various search forms
failing to return results when the username field contains trailing
whitespace (inserted by autocomplete).
* Fix tests to run the searches for real instead of mocking everything out.
* Fix SavedSearch.populate to only use the read only database in
production because in breaks things in tests. Specifically:
the posts get created in one db connection but searched for in
another, but the second transaction doesn't see the uncommitted posts
in the first transaction, so the search doesn't work.
