search: try to optimize slow searches.

Try to optimize certain types of common slow searches:

* Searches for mutually-exclusive tags (e.g. `1girl multiple_girls`,
  `touhou solo -1girl -1boy`)
* Relatively large tags that are heavily skewed towards old posts
  (e.g. lucky_star, haruhi_suzumiya_no_yuuutsu, inazuma_eleven_(series),
  imageboard_desourced).
* Mid-sized tags in the <30k post range that Postgres thinks are
  big enough for a post id index scan, but a tag index scan is faster.

The general pattern is Postgres not using the tag index because it
thinks scanning down the post id index would be faster, but it's
actually much slower because it degrades to a full table scan. This
usually happens when Postgres thinks a tag is larger or more common than
it really is. Here we try to force Postgres into using the tag index
when we know the search is small.

One case that is still slow is `2girls -multiple_girls`. This returns no
results, but we can't know that without searching all of `2girls`. The
general case is searching for `A -B` where A is a subset of B and A and B
are both large tags.

Hopefully fixes #581, #654, #743, #1020, #1039, #1421, #2207, #4070,
 #4337, #4896, and various other issues raised over the years regarding
slow searches.
This commit is contained in:
evazion
2021-10-11 23:39:03 -05:00
parent 0b22e873c9
commit f6abf39ebc
4 changed files with 62 additions and 15 deletions

View File

@@ -318,6 +318,13 @@ module Danbooru
[]
end
# Tag searches with less than this many results will be considered "small
# searches" and optimized specially. This is unnecessary unless you have a
# Danbooru-sized database.
def small_search_threshold
nil
end
# Your Pixiv PHPSESSID cookie. Get this by logging in to Pixiv and using
# the devtools to find the PHPSESSID cookie. This is need for Pixiv upload
# support.