Fix `Relation passed to #and must be structurally compatible. Incompatible values: [:joins] (ArgumentError)`
exception in `ordfav:evazion ratio:4:3` search. Broken by e849d8f1c.
We were effectively doing this:
q1 = Post.joins(:favorites, :media_asset).where("favorites.user_id = ?", 52664).order("favorites.id DESC")
q2 = Post.joins(:media_asset, :favorites).where("ROUND(media_assets.image_width::numeric / media_assets.image_height::numeric, 2) = 1.33")
q3 = q1.and(q2)
This failed because Rails didn't like the fact that the joins were in a different order when the
queries were `and`-ed together.
When searching posts by width, height, file size, or file extension, use the
values from the media_assets table rather than the posts table.
This makes filetype: searches faster because the file_ext is indexed on
the media assets table, but not on the posts table.
This paves the way for getting rid of the width, height, file_size, and
file_ext indexes on the posts table in the future. It's wasteful to
index these columns on both the posts table and the media assets table.
Fix a bug where searching for a negated tag inside the modqueue would show
active posts.
The bug was that in a search like this:
Post.in_modqueue.user_tag_match("-solo")
The `in_modqueue` condition would get sucked inside the tag search and negated
when we tried to apply the negation operator to the "solo" tag. So effectively
the `in_modqueue` condition would get negated and we would end up searching for
everything not in the modqueue.
Fix the order dropdown box on the modqueue page not working when filtering by tag.
This happened because when you do a tag search, the default order is set to `ORDER BY posts.id DESC`.
When you applied another order with the dropdown box, the new order would be tacked on to the old
ordering as a tiebreaker instead of replacing it, producing e.g. `ORDER BY posts.id DESC, queued_at DESC`
instead of `ORDER BY queued_at DESC`. The default order would always win because `posts.id` is
unique and doesn't have ties.
The fix is to have orders always override the previous order instead of adding to it.
Note that now if you use an `order:`, `ordfav:`, or `ordpool:` metatag in the search box on the
modqueue page, they will always be ignored in favor of the dropdown box.
Factor out the code that parses metatag values (e.g. `score:>5`) and
search URL params (e.g. `search[score]=>5`) into a RangeParser class.
Also fix a bug where, if the `search[order]=custom` param was used
without a `search[id]` param, an exception would be raised. Fix another
bug where if an invalid `search[id]` was provided, then the custom order
would be ignored and the search would be returned with the default order
instead. Now if you use `search[order]=custom` without a valid
`search[id]` param, the search will return no results.
Fix the search `flagger:evazion user:evazion` not returning the user's own self-flagged uploads.
Followup to a6e0872ce.
Fixes#4690: user profile 'flags' count links to /post_flags with different search criteria
* Fix it so non-moderators can't search deleted comments using the
`updater`, `body`, `score`, `do_not_bump_post`, or `is_sticky` fields.
Searching for these fields will exclude deleted comments.
* Fix it so non-moderators can search for their own deleted comments using the
`creator` field, but not for deleted comments belonging to other users.
* Fix it so that if a regular user searches `commenter:<username>`, they
can only see posts with undeleted comments by that user. If a moderator or
the commenter themselves searches `commenter:<username>`, they can see all
posts the user has commented on, including posts with deleted comments.
* Fix it so the comment count on user profiles only counts visible
comments. Regular users can only see the number of undeleted comments
a user has, while moderators and the commenter themselves can see the
total number of comments.
Known issue:
* It's still possible to order deleted comments by score, which can let
you infer the score of deleted comments.
Add a database model for storing AI-predicted tags, and add a UI for browsing and searching these tags.
AI tags are generated by the Danbooru Autotagger (https://github.com/danbooru/autotagger). See that
repo for details about the model.
The database schema is `ai_tags (media_asset_id integer, tag_id integer, score smallint)`. This is
designed to be as space-efficient as possible, since in production we have over 300 million
AI-generated tags (6 million images and 50 tags per post). This amounts to over 10GB in size, plus
indexes.
You can search for AI tags using e.g. `ai:scenery`. You can do `ai:scenery -scenery` to find posts
where the scenery tag is potentially missing, or `scenery -ai:scenery` to find posts that are
potentially mistagged (or more likely where the AI missed the tag).
You can browse AI tags at https://danbooru.donmai.us/ai_tags. On this page you can filter by
confidence level. You can also search unposted media assets by AI tag.
To generate tags, use the `autotag` script from the Autotagger repo, something like this:
docker run --rm -v ~/danbooru/public/data/360x360:/images ghcr.io/danbooru/autotagger ./autotag -c -f /images | gzip > tags.csv.gz
To import tags, use the fix script in script/fixes/. Expect a Danbooru-size dataset to take
hours to days to generate tags, then 20-30 minutes to import. Currently this all has to be done by hand.
This reverts commit 80ced3e418.
This turned out to be intentional. Rounding the aspect ratio to 2
decimal places is so that searches for exact ratios like `ratio:16:9` or
`ratio:1.78` work even when the ratio doesn't exactly match. Rounding to
2 decimal places means that the ratio: metatag has a 1% error tolerance.
Fix the ratio: metatag sometimes including wrong results due to rounding
errors. For example, searching for `ratio:>=4.0` would include post
3220414, which has an aspect ratio of 3.99879. This would get rounded up
to 2 decimal places to 4.00.
Refactor ratings to not be hardcoded in various places. Make it so
all ratings are defined in Post::RATINGS.
Also make it so that you can search multiple ratings at once with `rating:q,e`.
This setting automatically added the `-status:deleted` metatag to all searches. This meant deleted
posts were filtered out at the database level, rather than at the html level. This way searches
wouldn't have less-than-full pages.
The cost was that searches were slower, mainly because post counts weren't cached. Normally when you
search for a tag, we can get the post count from the tags table. If the search is actually like
`touhou -status:deleted`, then we don't know the count and we have to calculate it on demand.
This option is being removed because it did the opposite of what people thought it did. People
thought it made deleted posts visible, when actually it made them more hidden.
Factor out most of the tag edit logic from the Post class to a new
PostEdit class. The PostEdit class contains the logic for parsing tags
and metatags from the tag edit string, and for determining which tags
were added or removed by the edit.
Fixes various bugs caused by not calculating the set of added or removed
tags correctly, for example when tag category prefixes were used (e.g.
`copy:touhou`) or when the same tag was added and removed in the same
edit (e.g. `touhou -touhou`).
Fixes#5123: Tag categorization prefixes bypass deprecation check
Fixes#5126: Negating a deprecated tag will still cause the warning to show
Fixes#3477: Remove tag validator triggering on tag category changes
Fixes#4848: newpool: metatag doesn't parse correctly
Raise an error if the search is invalid for one of the following reasons:
* It contains multiple conflicting order: metatags (e.g. `order:score order:favcount` or `ordfav:a ordfav:b`).
* It contains a metatag that can't be used more than once: (e.g. `limit:5 limit:10`, `random:5 random:10`).
* It contains a metatag that can't be negated (e.g. `-order:score`, `-limit:20`, or `-random:20`).
* It contains a metatag that can't be used in an OR clause (e.g. ` touhou or order:score`, `touhou or limit:20`, `touhou or random:20`).
Switch to the post search engine using the new PostQuery parser. The new
engine fully supports AND, OR, and NOT operators and grouping expressions
with parentheses.
Highlights:
New OR operator:
* `skirt or dress` (same as `~skirt ~dress`)
Tags can be grouped with parentheses:
* `1girl (skirt or dress)`
* `(blonde_hair blue_eyes) or (red_hair green_eyes)`
* `~(blonde_hair blue_eyes) ~(red_hair green_eyes)` (same as above)
* `(pantyhose or thighhighs) (black_legwear or brown_legwear)`
* `(~pantyhose ~thighhighs) (~black_legwear ~brown_legwear)` (same as above)
Metatags can be OR'd together:
* `user:evazion or fav:evazion`
* `~user:evazion ~fav:evazion`
Wildcard tags can combined with either AND or OR:
* `black_* white_*` (find posts with at least one black_* tag AND one white_* tag)
* `black_* or white_*` (find posts with at least one black_* tag OR one white_* tag)
* `~black_* ~white_*` (same as above)
See 4c7cfc73 for more syntax examples.
Fixes#4949: And+or search?
Fixes#5056: Wildcard searches return unexpected results when combined with OR searches
Fix regression in 1ad0e8688. Caused by `relation.order_values` returning
an array of Arel nodes instead of an array of strings when doing a
`random:1` search.
Fix certain searches timing out when using sequential navigation (page=b1234).
The problem was that the so-called "small search optimization" (AKA: force Postgres
to use the tag index for small searches instead a sequential scan) wasn't triggering
because the ORDER BY clause for sequential navigation was `posts.id desc`, and we
were only checking for `posts.id DESC`.
Fix the /posts index controller not logging the normalized search query
to NewRelic when the search failed, either because of a tag limit error,
a search timeout, or a RSS feed rate limit error.
Also don't log the number of search results when it's an API request or
failed search. This is to avoid doing a potentially slow full post count
when it's not otherwise needed.
Make the `order:random` metatag truly randomize the search. Add a
`random:N` metatag that returns up to N random posts, like what
`order:random` did before.
`order:random` now returns the entire search in random order. Before it
just returned a pageful of pseudorandom posts. This will be more
accurate for small searches, but slower for large searches. If
`order:random` times out, try `random:N` instead.
The `random:N` metatag returns up to N pseudorandom posts. This is
faster than `order:random` for large searches, but for small searches,
it may return less than N posts, and the randomness may be biased. Some
posts may be more likely than others to appear. N must be between 0 and
200.
Also, `/posts?tags=touhou&random=1` now redirects to `/posts?tags=touhou+random:N`.
Before the `random=1` param acted like a free `order:random` tag; now it
redirects to a `random:N` search, so it counts against your tag limit.
Optimize metatag searches involving usernames, including user:,
approver:, appealer:, commenter:, upvoter:, etc.
Do `User.find_by_name` instead of `User.name_matches` because this
fetches the user upfront instead of doing it inside a subquery. Using a
subquery makes the SQL more complicated and leads to worse query plans.
This especially helps searches involving multiple username metatags.
Allow admins to remove votes on posts. This is for fixing vote abuse.
Votes can be removed by going to the vote list on the /post_votes page,
or by clicking on a post's score, then using the "Remove" option in the
"..." dropdown menu next to the vote.
Votes are soft-deleted - they're marked as deleted in the database, but
not fully deleted. Removed votes are only visible to admins, not to
regular users. When a vote is removed by an admin, it leaves a mod
action.
Technically it's possible to undelete votes, but there's no UI for it.
Add `upvotes:N`, `downvotes:N`, `order:upvotes`, `order:downvotes`,
`order:upvotes_asc`, `order:downvotes_asc` metatags.
In the API, the field is called up_score / down_score. Here it's called
`upvotes` and `downvotes` because this should be easier to understand
for end users.
Note that internally, `down_score` is negative. A post that matches
`downvotes:>5` will have down_score < -5 internally.
* Change `age:` metatag to require time units. This means e.g.
`age:<600` no longer works; instead you have to say `age:<600sec`.
* Allow time units in the `age:` metatag to be abbreviated as long as
they're unambiguous. This means `age:<60sec`, `age:<5min`, and
`age:<5mon` now work, in addition to `age:<60s` and `age:<60seconds`.
* Allow the `ratio:` metatag to be written like `ratio:16/9` in addition
to `ratio:16:9`.
* Fix invalid date searches like `date:foo` or `date:05-15-2021`
to return nothing instead of raising an "undefined method
'beginning_of_day' for nil" exception. (`date:05-15-2021` is invalid
because it's parsed as DD-MM-YYYY).
* Fix invalid searches like `score:foo`, `ratio:foo`, and `mpixels:foo`
to return nothing instead of being treated like `score:0`, `ratio:0`,
`mpixels:0`.
* Fix `age:<60m` to return nothing instead of silently being treated
like `age:<60seconds`.
* Fix `age:foo` to return nothing instead of silently being treated like
`age:0d` (return all uploads from today).
Fixes#4389.
Show the length of videos and animated posts in the thumbnail. The
length is shown the top left corner in MM:SS format. This replaces the
play button icon.
Show a speaker icon instead of a music note icon for posts with sound.
Doing this requires doing `.includes(:media_asset)` in a bunch of
places to avoid N+1 queries when we access the post's duration.
Move all the code for defining tag categories from the config file to
TagCategory. It didn't belong in the config because it's not possible to
add new tag categories purely in the config without editing other things
like the CSS.
Also change it so that tag colors are hardcoded in the CSS instead of
generated using ERB. Generating the CSS in ERB meant that the Docker
build had to recompile the CSS on every commit, even when it didn't
change, because it relied on Ruby code outside the CSS that we couldn't
guarantee didn't change.
Try to optimize certain types of common slow searches:
* Searches for mutually-exclusive tags (e.g. `1girl multiple_girls`,
`touhou solo -1girl -1boy`)
* Relatively large tags that are heavily skewed towards old posts
(e.g. lucky_star, haruhi_suzumiya_no_yuuutsu, inazuma_eleven_(series),
imageboard_desourced).
* Mid-sized tags in the <30k post range that Postgres thinks are
big enough for a post id index scan, but a tag index scan is faster.
The general pattern is Postgres not using the tag index because it
thinks scanning down the post id index would be faster, but it's
actually much slower because it degrades to a full table scan. This
usually happens when Postgres thinks a tag is larger or more common than
it really is. Here we try to force Postgres into using the tag index
when we know the search is small.
One case that is still slow is `2girls -multiple_girls`. This returns no
results, but we can't know that without searching all of `2girls`. The
general case is searching for `A -B` where A is a subset of B and A and B
are both large tags.
Hopefully fixes#581, #654, #743, #1020, #1039, #1421, #2207, #4070,
#4337, #4896, and various other issues raised over the years regarding
slow searches.
When a search is performed, we cache the post count so we don't have to
calculate it again every time the user switches pages. However, if the
count times out, we didn't cache it before, causing us to do a slow
count on every page load. This usually happens on multi-tag searches
that return a lot of results, `1girl solo` for example.
This changes it so that the count is cached even when it times out. This
will speed up large multi-tag searches.
This also changes it so that the count is cached for a fixed 5 minutes.
Before it was variable based on the size of the count, but this probably
didn't make much difference.
Use the `string_to_array(tag_string, ' ')` index instead of the
`tag_index` for tag searches. The string_to_array index lets us treat
the tag_string as an array for searching purposes. This lets us get rid
of the tag_index column and the test_parser dependency in the future.
Stop updating the fav_string attribute on posts. The column still exists
on the table, but is no longer used or updated.
Like the pool_string in 7d503f08, the fav_string was used in the past to
facilitate `fav:X` searches. Posts had a hidden fav_string column that
contained a list of every user who favorited the post. These were
treated like fake hidden tags on the post so that a search for `fav:X`
was treated like a tag search.
The fav_string attribute has been unused for search purposes for a while
now. It was only kept because of technicalities that required
departitioning the favorites table first (340e1008e) before it could be
removed. Basically, removing favorites with `@favorite.destroy` was
slow because Rails always deletes object by ID, but we didn't have an
index on favorites.id, and we couldn't easily add one until the
favorites table was departitioned.
Fixes#4652. See https://github.com/danbooru/danbooru/issues/4652#issuecomment-754993802
for more discussion of issues caused by the fav_string (in short: write
amplification, post table bloat, and favorite inconsistency problems).
Optimize counting the number of posts returned by fav:<name> and
pool:<name> searches. Use cached counts to avoid slow count(*) queries
for users with lots of favorites.
