Fix a severe performance regression on the posts/index page introduced
by 6ca42947.
Short answer: scan_query dynamically allocated a regex inside an
inner loop that was called thousands of times per pageload.
Long answer:
* The post index page checks each post to see if they're tagged loli/shota,
* This triggers a call to Post#tag_array for every post.
* Post#tag_array called scan_query to split the tag string.
* scan_query loops over the tag string, checking if each tag matches the
regex /#{METATAGS.join("|")}:/.
* This regex uses string interpolation, which makes Ruby treat as a
dynamic value rather than a static value. Ruby doesn't know the
interpolation is static here. This causes the regex to be reallocated
on every iteration of the loop, or in other words, for every tag in
the tag string.
* This caused us to do thousands of regex allocations per pageload. On
average, a posts/index pageload contains 20 posts with ~35 tags per
post, or 7000+ total tags. Doing this many allocations killed performance.
The fix:
* Don't use scan_query for Post#tag_array. We don't have to fully parse
the tag_string here, we can use a simple split.
* Use the /o regex flag to tell Ruby to treat the regex as static and
only evaluate the interpolation once.
* Fix not being able to use the status: metatag twice in the same search.
* Fix status:active excluding banned posts.
* Fix status:garbage returning all posts.
Support exclusive ranges for numeric metatags. For example, `id:5...10`
is equivalent to `id:>=5 id:<10`. Useful for splitting searches into id
ranges without the endpoints overlapping: id:100...200, id:200...300,
id:300...400.
Support using the same numeric-valued metatag twice in the same search.
Numeric-valued metatags are those taking an integer, float, filesize, or
date argument. Previously using the same metatag twice would cause the
second metatag to overwrite the first metatag.
Examples:
* "id:>5 id:<10"
* "width:>500 width:<1000"
* "date:>2019-01-01 date:<2020-01-01"
Support using quoted values with all metatags. For example: user:"blah blah",
pool:"blah blah", commentary:"blah blah", etc. Things like rating:"safe",
id:"42" also work. Both single and double quotes are supported.
Also make the status: and rating: metatags fully free. Before only
status:deleted and rating:s were free.
Refactor so that all metatags that use parse_helper (which is most
integer-valued metatags) support the special "any" and "none" keywords.
This is mainly for consistency, since it's really only useful for using
width:none and height:none to find for certain old unsupported filetypes
that have null width/height values.
Fold parse_helper_fudged into parse_helper. This is used to make
mpixels:N and filesize:N do an inexact match with a 5% error tolerance.
For example, filesize:1mb is really equivalent to filesize:0.95mb..1.05mb
because an exact filesize:1mb search is unlikely to match anything.
Bug: extremely long sources cause the Tags column to become extremely
wide. Caused by the source link not having the word-break property set.
Fix: use post_source_tag, which lets the `a[rel=external] { word-break: break-word; }`
rule take effect.
Example: https://danbooru.donmai.us/post_versions?search%5Bpost_id%5D=3809742
Bug: searching for "filetype:jpg -user:evazion" would negate the
filetype:jpg metatag too. This was because the -user:<name> metatag was
negating the entire cumulative relation instead of just the user:<name>
clause.
Add:
* commentary:true (posts with commentary)
* commentary:false (posts without commentary)
* commentary:translated (posts with translated commentary)
* commentary:untranslated (posts with untranslated commentary)
* commentary:"text" (posts where any commentary field matches "text")
Known issues:
* There's no way to escape the true, false, translated, or
untranslated keywords to do a literal text search for commentaries
containing one of these keywords.
* Negated searches may be slow. Using a left outer join instead of a
subquery would be faster in most cases, but negating it is harder.
* Fix fav:<user> searches to return no results instead of raising a
UserPrivilege error when the user has private favorites.
* Fix fav:<nonexistent_user> raising a UserPrivilege error instead of
returning no results.
* Fix -ordfav:<user> not being supported.
Change favgroup:<name> searches to return no results instead of raising
a UserPrivilege error when an unpermitted user searches for a private
favgroup.
Partial fix for #4389.
Partial fix for #4389.
* Fix invalid username searches returning all posts instead of no posts.
* Fix "user:A user:B" returning results for user:B instead of no results.
* Fix "approver:A approver:B" returning results for approver:B instead of no results.
* Add support for negated -commenter, -noter, -noteupdater, -upvote, -downvote metatags.
* Add support for "any" and "none" values for all username metatags,
including negated metatags that didn't support "any" or "none" before.
* Change noter:any and commenter:any to include posts with deleted notes
or comments. Note that commenter:<username> already included deleted
comments before. This is so that commenter:any has the same behavior
as commenter:<username>
* Fix corrupted image detection. We were shelling out to vips and trying
to grep for error messages, but the error message for jpeg files changed.
Now we load the file in ruby vips, which raises an error on failure.
* Don't attempt to redownload corrupted images. If a download completes
without any errors yet the downloaded file is corrupt, then something is
wrong at the source and redownloading is unlikely to help. Let the
upload fail and the user retry if necessary.
* Validate that all uploads are uncorrupted, including files uploaded
from a computer, not just files uploaded from a source.
Fix table columns being collapsed to one character wide in certain
cases. Caused by setting `word-break: break-word` on div.prose, which
was needed to stop very long unbroken strings making columns too wide,
but has the opposite effect of making columns too narrow in other cases.
Example: https://danbooru.donmai.us/forum_topics/13868?page=28#forum_post_164950
* Make IP bans soft deletable.
* Add a hit counter to track how many times an IP ban has blocked someone.
* Add a last hit timestamp to track when the IP ban last blocked someone.
* Add a new type of IP ban, the signup ban. Signup bans restrict new
signups from editing anything until they've verified their email
address.
* Call them hidden tags instead of censored tags so that users don't
think that buying gold removes censorship from images (yes, this happened).
* Remove API ratelimit information (not relevant to most users).
Use validations instead of raising an exception when the password is
incorrect so that the controller can display errors sensibly.
Also fix users being logged out even when the deletion attempt failed
due to an incorrect password.
Remove the 10-try limit when there's a name conflict during renaming. We
forgot to increment the loop counter so this did nothing. This wasn't
necessary anyway since the loop will always terminate eventually because
names have finite length.
Name changes for deleted users are already visible to mods, so the only
thing we need to do here is to generate a name change before the user is
deleted.
* Let moderators see name changes for deleted users on the user name
change requests index and show pages. Before they could see name changes
for deleted users on user profiles, but not on the user name changes index.
* Let members see previous names on profile pages. Before they could see
previous names on the user name changes index, but not on profile pages
(ref: #4382).
* Allow tagging a post with a `disapproved:<disinterest|breaks_rules|poor_quality>` to disapprove it.
* Disallow disapproving active posts.
Fixes#4384.
Fix exception when submitting an upload and an in-progress preprocessed
upload already exists. In this case we forgot to pass the upload params
when calling UploadService#delayed_start.
Setting max-width causes notes to overflow the note body if they set a
fixed `width` or they use `white-space: nowrap` and the line is wider
than the max-width. Using `width: min-content` instead makes the note
shrink to fit but doesn't prevent it from expanding if it needs to.
If the image is blacklisted on initial page load then unblacklisted
after the image is loaded, then the notes will be mispositioned at the
bottom of the image. This is because we relied on $image.height() to
calculate the note position, but the image height is zero when the image
is hidden.
Potential fix for #4370.
The file url in the data attributes on the image container doesn't
contain the tags for performance reasons. The post data attributes are
used for thumbnails too and we don't want to include tagged filenames
for those because it requires looking up tag categories, which triggers
a lot of redis calls on the post index page.
* Fix preview images not being resized to fit the screen when using the
bookmarklet.
* Fit images to both screen width and height by default. Previously we
fit to screen width only, so tall images were hard to see.
* Allow clicking on the image or pressing Z to toggle image size.
* Move size information to above the image and add resize links:
** small: fit width and height
** large: fit width
** full: fit none
* Bind the image error handler in an onerror attribute on the image
itself so that it will always fire. Before it wouldn't fire if the image
failed immediately on pageload before we could bind the error handler.
evazion
evazion