Make PostQueryBuilder apply aliases earlier, immediately after parsing
the search.
On the post index page there are multiple places where we need to apply
aliases:
* When running the search with PostQueryBuilder#build.
* When calculating the search count with PostQueryBuilder#fast_count.
* When calculating the related tags for the sidebar.
* When tracking missed searches and popular searches for Reportbooru.
* When looking up wiki excerpts.
Applying aliases after parsing ensures we only have to apply aliases
once for all of these things.
We also normalize the order of tags in searches and strip repeated tags.
This is so that we have consistent cache keys for fast_count.
* Fixes searches for aliased tags being counted as missed searches (fixes#4433).
* Fixes wiki excerpts not showing up when searching for aliased tags.
When doing a tag search, we have to be careful about which user we're
running the search as because the results depend on the current user.
Specifically, things like private favorites, private favorite groups,
post votes, saved searches, and flagger names depend on the user's
permissions, and whether non-safe or deleted posts are filtered out
depend on whether the user has safe mode on or the hide deleted posts
setting enabled.
* Refactor internal searches to explicitly state whether they're
running as the system user (DanbooruBot) or as the current user.
* Explicitly pass in the current user to PostQueryBuilder instead of
implicitly relying on the CurrentUser global.
* Get rid of CurrentUser.admin_mode? (used to ignore the hide deleted
post setting) and CurrentUser.without_safe_mode (used to ignore safe
mode).
* Change the /counts/posts.json endpoint to ignore safe mode and the
hide deleted posts settings when counting posts.
* Fix searches not correctly overriding the hide deleted posts setting
when multiple status: metatags were used (e.g. `status:banned status:active`)
* Fix fast_count not respecting the hide deleted posts setting when the
status:banned metatag was used.
* Use libvips instead of ruby-imagespec for reading dimensions of jpeg, png, and gif files.
* Copy the code for reading the dimensions of flash files from ruby-imagespec.
Fixes an incompatibility between ruby-imagespec and the rubocop gem that
prevented us from including rubocop in the Gemfile.
* Add MediaFile abstraction. A MediaFile represents an image or video file.
* Move filetype detection and dimension parsing code from uploads to MediaFile.
Don't show "You have already hidden this post" and "Post is already
active" errors when disapproving a post. Just hide the post anyway
regardless of whether we literally created a post disapproval record.
Add <link rel="prev"> and <link rel="next"> tags to the forum topic show
page, the pool show page, and the favgroup show page.
This is kind of useless now since Google hasn't used these tags for SEO
for years ([1]), and userscripts can use `#paginator-next` and
`#paginator-prev` to find the next/prev pages, but whatever.
[1]: https://yoast.com/google-doesnt-use-rel-prev-next-for-pagination
On the post index page, show the wiki excerpt if the search includes a
single tag, even if the tag is negated or the search includes other
metatags.
If the search includes a single pool: or ordpool: metatag, show the pool
excerpt even if the search includes other metatags.
* Add unaliased:<tag> metatag. This allows you to search for a tag
without applying aliases. This is mainly useful for debugging purposes
and for searching for large tags that are in the process of being
aliased but haven't had all their posts moved yet.
* Remove the "raw" url param from the posts index page. The "raw" param
also caused the search to ignore aliases, but it was undocumented and
exploitable. It was possible to use the raw param to view private
favorites since favorites are treated like a hidden tag.
Forgot to account for negated metatags in normalize_query after e987f070.
Fixes a bug where wrong page counts were displayed for searches
involving negated metatags due to incorrect query normalization.
Fix not being able to negate the following metatags:
* id (didn't support ranges)
* md5
* width
* height
* mpixels
* ratio
* score
* favcount
* filesize
* date
* age
* tagcount
* pixiv
Bug: in some cases searching for multiple metatags would cause one
metatag to be ignored. For example, a search for {{user:1 pool:2}} would
be treated as a search for {{pool:2}}.
Cause: we used `ActiveRecord::Relation#merge` to combine two relations,
which was wrong because `merge` doesn't combine `column IN (?)` clauses
correctly. If there are two `column IN (?)` clauses on the same column,
then `#merge` takes only the second clause and ignores the first.
Fix: write our own half-baked `#and` method to work around Rails'
broken-by-design `#merge` method.
ref: https://github.com/rails/rails/issues/33501.
`normalize_query` is used in certain places on the post index page where
we don't want to pay the cost of looking up tag aliases (namely inside
fast_count, in post_search_count_js, and in tag change notices). Don't
normalize aliases by default unless we need to.
Several fixes for the "This tag is under discussion" notice on the post
index page:
* Fix the notice appearing for BURs that aren't pending.
* Fix the notice never going away because of the cache never expiring.
* List all topics when a tag is involved in multiple BURs.
* Link to the forum post instead of the forum topic (fix#4421).
* Optimization: don't check for BURs when the search isn't a simple
single tag search.
* Add a `tags` field to the bulk update requests table for tracking all
tags involved in the request (excluding tags in mass updates that are
negated/optional/wildcards). Known issue: doesn't handle tag type
prefixes in mass updates correctly (e.g. `mass update foo -> artist:bar`
doesn't detect the tag `bar`).
* Allow searching the /bulk_update_requests page by tags.
We don't really need to cache the notice here, but we do it anyway to
reduce queries on the post index page.
Don't use pretty names (spaces instead of underscores) for pools in post
tooltips. This is for consistency with tags (which have underscores
here) and for easier copy & pasting.
Treat the following searches as literal text searches instead of as
special keywords:
* source:none
* commentary:true
* commentary:false
* commentary:translated
* commentary:untranslated
* Eliminate the `parse_query` method.
* Move all the metatag handling logic from the `build` method
to `metatag_matches` and helper methods.
This is to get all the main metatag handling logic in one place, inside
`metatag_matches`, so that it's easier to add new metatags and to handle
things like negated metatags more consistently.
Fix order:custom not working. Also change order:custom to return no
posts under the following error conditions:
* {{order:custom}} (id metatag isn't present)
* {{id:42 order:custom}} (id metatag isn't a list)
* {{id:>42 order:custom}} (id metatag isn't a list)
* {{id:1,2 id:2,3 order:custom}} (id metatag is present twice)
Bug: If a Member had the hide_deleted_posts option turned on and did a
two tag search, no pages would show up.
Cause: The hide_deleted_posts option implicitly adds the -status:deleted
tag, but this tag wasn't considered a free metatag, so this caused
Post.fast_count to fail and return zero because the search was treated
as a three tag search.
ref: https://danbooru.donmai.us/forum_topics/16829
* Move various search parser helper methods (`has_metatag?`,
`is_single_tag?` et al) from PostSets and the Tag model to
PostQueryBuilder.
* Fix various minor bugs stemming from trying to check if a search query
contains certain metatags using regexes or other adhoc techniques.
* Make scan_query, parse_query, normalize_query into instance methods
instead of class methods. This is to a) clean up the API and b)
prepare for moving certain tag utility methods into PostQueryBuilder.
* Fix a few cases where a caller used scan_query when they should have
used split_query or parse_tag_edit.
* Support negating the child: and embedded: metatags.
* Fix approver:<any|none>, disapproved:<reason>, commentary:<type> being
case sensitive.
* Fix child:garbage, locked:garbage, embedded:garbage returning all
posts instead of no posts.
* Fix not being able to use source:, locked:, or -id: twice in the same
search.
Fix a severe performance regression on the posts/index page introduced
by 6ca42947.
Short answer: scan_query dynamically allocated a regex inside an
inner loop that was called thousands of times per pageload.
Long answer:
* The post index page checks each post to see if they're tagged loli/shota,
* This triggers a call to Post#tag_array for every post.
* Post#tag_array called scan_query to split the tag string.
* scan_query loops over the tag string, checking if each tag matches the
regex /#{METATAGS.join("|")}:/.
* This regex uses string interpolation, which makes Ruby treat as a
dynamic value rather than a static value. Ruby doesn't know the
interpolation is static here. This causes the regex to be reallocated
on every iteration of the loop, or in other words, for every tag in
the tag string.
* This caused us to do thousands of regex allocations per pageload. On
average, a posts/index pageload contains 20 posts with ~35 tags per
post, or 7000+ total tags. Doing this many allocations killed performance.
The fix:
* Don't use scan_query for Post#tag_array. We don't have to fully parse
the tag_string here, we can use a simple split.
* Use the /o regex flag to tell Ruby to treat the regex as static and
only evaluate the interpolation once.
* Fix not being able to use the status: metatag twice in the same search.
* Fix status:active excluding banned posts.
* Fix status:garbage returning all posts.
Support exclusive ranges for numeric metatags. For example, `id:5...10`
is equivalent to `id:>=5 id:<10`. Useful for splitting searches into id
ranges without the endpoints overlapping: id:100...200, id:200...300,
id:300...400.
Support using the same numeric-valued metatag twice in the same search.
Numeric-valued metatags are those taking an integer, float, filesize, or
date argument. Previously using the same metatag twice would cause the
second metatag to overwrite the first metatag.
Examples:
* "id:>5 id:<10"
* "width:>500 width:<1000"
* "date:>2019-01-01 date:<2020-01-01"
Support using quoted values with all metatags. For example: user:"blah blah",
pool:"blah blah", commentary:"blah blah", etc. Things like rating:"safe",
id:"42" also work. Both single and double quotes are supported.
Also make the status: and rating: metatags fully free. Before only
status:deleted and rating:s were free.
Refactor so that all metatags that use parse_helper (which is most
integer-valued metatags) support the special "any" and "none" keywords.
This is mainly for consistency, since it's really only useful for using
width:none and height:none to find for certain old unsupported filetypes
that have null width/height values.
Fold parse_helper_fudged into parse_helper. This is used to make
mpixels:N and filesize:N do an inexact match with a 5% error tolerance.
For example, filesize:1mb is really equivalent to filesize:0.95mb..1.05mb
because an exact filesize:1mb search is unlikely to match anything.
Bug: extremely long sources cause the Tags column to become extremely
wide. Caused by the source link not having the word-break property set.
Fix: use post_source_tag, which lets the `a[rel=external] { word-break: break-word; }`
rule take effect.
Example: https://danbooru.donmai.us/post_versions?search%5Bpost_id%5D=3809742
Bug: searching for "filetype:jpg -user:evazion" would negate the
filetype:jpg metatag too. This was because the -user:<name> metatag was
negating the entire cumulative relation instead of just the user:<name>
clause.
Add:
* commentary:true (posts with commentary)
* commentary:false (posts without commentary)
* commentary:translated (posts with translated commentary)
* commentary:untranslated (posts with untranslated commentary)
* commentary:"text" (posts where any commentary field matches "text")
Known issues:
* There's no way to escape the true, false, translated, or
untranslated keywords to do a literal text search for commentaries
containing one of these keywords.
* Negated searches may be slow. Using a left outer join instead of a
subquery would be faster in most cases, but negating it is harder.
* Fix fav:<user> searches to return no results instead of raising a
UserPrivilege error when the user has private favorites.
* Fix fav:<nonexistent_user> raising a UserPrivilege error instead of
returning no results.
* Fix -ordfav:<user> not being supported.
Change favgroup:<name> searches to return no results instead of raising
a UserPrivilege error when an unpermitted user searches for a private
favgroup.
Partial fix for #4389.
Partial fix for #4389.
* Fix invalid username searches returning all posts instead of no posts.
* Fix "user:A user:B" returning results for user:B instead of no results.
* Fix "approver:A approver:B" returning results for approver:B instead of no results.
* Add support for negated -commenter, -noter, -noteupdater, -upvote, -downvote metatags.
* Add support for "any" and "none" values for all username metatags,
including negated metatags that didn't support "any" or "none" before.
* Change noter:any and commenter:any to include posts with deleted notes
or comments. Note that commenter:<username> already included deleted
comments before. This is so that commenter:any has the same behavior
as commenter:<username>
* Fix corrupted image detection. We were shelling out to vips and trying
to grep for error messages, but the error message for jpeg files changed.
Now we load the file in ruby vips, which raises an error on failure.
* Don't attempt to redownload corrupted images. If a download completes
without any errors yet the downloaded file is corrupt, then something is
wrong at the source and redownloading is unlikely to help. Let the
upload fail and the user retry if necessary.
* Validate that all uploads are uncorrupted, including files uploaded
from a computer, not just files uploaded from a source.
Fix table columns being collapsed to one character wide in certain
cases. Caused by setting `word-break: break-word` on div.prose, which
was needed to stop very long unbroken strings making columns too wide,
but has the opposite effect of making columns too narrow in other cases.
Example: https://danbooru.donmai.us/forum_topics/13868?page=28#forum_post_164950
evazion
evazion