Switch the ActiveJob backend from DelayedJob to GoodJob. Differences:
* The job worker is run with `bin/good_job start` instead of `bin/delayed_job`.
* Jobs have an 8 hour timeout instead of a 4 hour timeout.
* Jobs don't automatically retry on failure.
* Finishing jobs are preserved and pruned after 7 days.
This is the first step towards replacing DelayedJob with GoodJob. Compared to
DelayedJob:
* GoodJob supports Rails 7 (DelayedJob is currently a blocker for Rails 7
because it has a version bound on ActiveRecord <6.2).
* GoodJob has a builtin admin dashboard.
* GoodJob supports threaded job workers.
* GoodJob supports scheduled cronjobs.
* GoodJob supports healthchecks for workers.
* GoodJob uses Postgres notifications instead of polling to pick up new
jobs. This allows jobs to be picked up faster and scales better with
large numbers of workers.
https://github.com/bensheldon/good_job
Fix an `undefined method 'find' for Maintenance::User:Module` exception
when disabling email notifications using the "Disable notifications"
link in dmails.
Add a fix script to fix email addresses that are invalid or that contain
common typos. If the email can't be fixed, usually because the fixed
address is already in use by another account, then the email address is
marked undeliverable.
Fix mailers to not attempt deliveries to invalid or nonexistent email
addresses. This usually happened when someone changed their email, and
we tried to send a confirmation email to a nonexistent address.
Fix an exception when checking out on Stripe caused by using the
`allow_promotion_codes` and `discounts` parameters at the same time.
Stripe::InvalidRequestError - You may only specify one of these parameters: allow_promotion_codes, discounts
Switch the Ruby memory allocator from Glibc malloc to Jemalloc. Jemalloc
supposedly uses less memory than Glibc malloc because it's better at
handling memory fragmentation. It also has detailed internal statistics
to help monitor allocator behavior.
We use the LD_PRELOAD method of loading Jemalloc instead of building it
into Ruby so that we can switch allocators at runtime.
Further micro-optimize thumbnails to reduce string allocations.
`Post#levelblocked?` gets called once per thumbnail. Before it split the
tag string, which meant one string allocation for each tag on each post.
This added up to thousands of string allocations per pageload.
Do a few micro-optimizations to reduce the number of memory allocations
during thumbnail generation.
This commit, combined with freezing string literals in a7dc05 and
67b961, reduces the number of allocations on the front page from 180,000
to 150,000, and the number of retained objects from 8,000 to 4,000.
Monkey-patch Symbol#to_s to return a frozen (immutable) string instead
of a mutable string.
This should reduce string allocations, and thereby reduce memory usage
and garbage collector pressure, but it may be incompatible with
libraries that expect Symbol#to_s to return a mutable string.
https://bugs.ruby-lang.org/issues/16150https://github.com/Shopify/symbol-fstring
* Mark /comments/:id links in comment timestamps as nofollow to prevent
Googlebot from crawling these links.
* Mark /posts/:id/show_seq links as disallowed in robots.txt to prevent
Googlebot from crawling forward/back links on posts.
* Increase the default thumbnail size from small (150x150) to medium (180x180).
* Change the mobile layout to use three posts per row instead of two for small thumbnails.
Parent/child posts are still 150x150 to avoid taking up even more space above posts.
For small thumbnails, use 180x180 thumbnails scaled down to 150x150.
This is so we can get rid of 150x150 images and just use 180x180 for
both small and medium size thumbnails.
Also fix RSS feeds, XML sitemaps, and Discord embeds to use 360x360
thumbnails instead of 150x150 thumbnails.
* Bottom-align thumbnails on desktop. This is so that vote buttons line
up beneath thumbnails. Also so that pool titles line up on the pool
gallery page. This is desktop only because it tends to leave large
gaps above thumbnails on mobile.
* Make thumbnails fixed-height so that the layout doesn't shift as
thumbnails are loaded.
Fix the related tags section being completely hidden when it was
collapsed. The `.hidden` CSS class that was used by related tags
conflicted with the `.hidden` utility class added in 8841de68ac.
Add a rate limit of 1 request per minute to the /explore/posts/popular.json endpoint.
This is to deal with a particular bot checking this page multiple times
per second.
Add a rate limit of 1 request per 2 seconds to the post RSS feed
endpoint (/posts.atom).
This lets you check your feeds 30 times per minute, or 1800 times per
hour. The previous limit was 10 requests per second.
This is because there are too many bad RSS feed reader bots constantly
checking the same tags over and over again, 24 hours a day, as fast
they can.
Refactor controllers so that endpoint rate limits are declared locally,
with the endpoint, instead of globally, in a single method in ApplicationController.
This way an endpoint's rate limit is declared in the same file as the
endpoint itself.
This is so we can add fine-grained rate limits for certain GET requests.
Before rate limits were only for non-GET requests.
Make "show scores" setting persistent.
The setting is stored in a `post_preview_show_votes` cookie. This means
it's remembered on a per-device basis, but not on a per-account basis.
This is so users without an account can use the setting, and so you can
use different settings on desktop and mobile.
The `view=score` URL param has been replaced by `show_votes=true`. The
`show_votes` URL param overrides the `post_preview_show_votes` cookie.
Make setting the thumbnail size persistent.
The setting is stored in a `post_preview_size` cookie. This cookie can
be overridden by the `size` URL param, like so:
https://danbooru.donmai.us/posts?tags=touhou&size=180
The `size` param is mainly for testing different sizes without setting a cookie.
Remove the ability to lock a tag's category. Before a moderator could
lock a tag such that only an admin could change the tag's category.
Nowadays the ability to change a tag's category is based on the tag's
size. Members can change tag categories for tags with up to 50 posts,
and Builders can change categories for tags with up to 1000 posts.
Manually locking tags is not necessary.
We only had a few dozen locked tags, mostly random *_(cosplay) tags or
company name tags. Most of these are holdovers from moderators randomly
locking tags like ten years ago.
The `is_locked` field is still in the database, so it is still returned
by the /tags.json API, even though it is unused.
Fix the tag edit page letting users attempt to change a tag's category,
even when the tag was too large for the user to change. Trying to change
the category would fail, but with a confusing error message.
If the `size` URL param wasn't present, then `size=null` would be passed
to `/posts/:id.js`, which would fail because `null` wasn't a valid size.
Regression in 8841de68ac.
Fix thumbnail CSS to not use `width: auto`, because that made the page
reflow. It made thumbnails start out at 0x0 size, then resize to the
actual size only after the image was loaded.
Make the posts/index template easier to read by putting everything in
one file, instead of splitting it up into a bunch of partials that
aren't used anywhere else.
Fix an error being raised when trying to generate thumbnails for corrupt
files. If the original image is corrupt, then ignore any errors and let
libvips try to generate a thumbnail as best it can. This will usually
result in an incomplete thumbnail.
This is used to provide higher resolution thumbnails for high pixel
density displays, such as phones or laptops. If your screen has a 2x
pixel density ratio, then 360x360 thumbnails will be rendered at 720x720
resolution.
We use WebP here because it's about 15% smaller than the equivalent
JPEG, and because if a device has a high enough pixel density to use
this, then it probably supports WebP.
720x720 thumbnails average about 36kb in size, compared to 20.35kb for
360x360 thumbnails and 7.55kb for 180x180 thumbnails.
Fix the /pools/gallery page layout being broken by 8841de68a.
This required refactoring the PostGalleryComponent to take a set of
PostPreviewComponents instead of a set of Posts.
The upshot is that it's technically possible to have adjustable
thumbnail sizes on the pool gallery page now (although this is not yet
exposed in the UI).
Disable cropped (square) thumbnails on mobile. Use regular uncropped
thumbnails instead.
This is for a few reasons:
* It made it harder to support multiple thumbnail sizes or file formats,
since we need a cropped and uncropped version for every size and format.
* The cropping algorithm wasn't that great and sometimes cropped out
important parts of the image.
* The thumbnail type (cropped or uncropped) was automatically chosen
clientside based on the user's screen size, which made certain things
like adjustable thumbnail sizes more difficult because we didn't know
which thumbnail type the user actually had.
This may return again as an option in the future, but for now it's disabled.
Three thumbnails per row was pretty tight for most phones since the most
common phone resolution is 360x640. Two thumbnails per row lets us have
thumbnails up to 180x180 in size.
Factor out thumbnail galleries into a PostGallery component.
This changes the html structure so that post galleries on all pages are
always wrapped in a `.posts-container` class. This fixes an issue with
thumbnails on the pool show page not being aligned correctly on mobile,
like they are on the post index page. This also affected thumbnail
galleries on other pages, like wiki pages and user profiles.
Remove the SFTP file storage backend. Downstream users can use either
sshfs (which is what Danbooru now uses in production) or rclone instead.
The Ruby SFTP gem was much slower than sshfs.
These other formats aren't actually generated during upload, but support
for creating them is there.
Also tune the parameters for generating JPEGs:
* Use Q=85 instead of Q=90 because Q=85 enables 4:2:0 chroma
subsampling, while Q=90 doesn't use subsampling. Subsampling reduces
filesize by ~30% in most cases. It does reduce quality for certain
images, particularly for images with lots of bright red, but in most
cases the quality difference isn't noticeable.
* Enable several MozJPEG-specific options, including trellis
quantization and scan optimization. These reduce filesize without
reducing quality, at the cost of slower encoding times.
Calculate the dimensions of thumbnails ourselves instead of letting
libvips calculate them for us. This way we know the exact size of
thumbnails, so we can set the right width and height for <img> tags. If
we let libvips calculate thumbnail sizes for us, then we can't predict
the exact size of thumbnails, because sometimes libvips rounds numbers
differently than us.
The problem was that we were stripping color profiles from thumbnails,
but we weren't setting `export_profile: "srgb"` to convert images to
sRGB first. This resulted in wrong colors for images with non-sRGB color
profiles, such as Adobe RGB.
The fix is to convert images to sRGB when possible, while leaving CMYK
and greyscale images alone. We leave CMYK images alone because we can't
convert CMYK to sRGB without losing color. We leave greyscale images
alone if they don't have a color profile, that way they stay as
one-channel greyscale (or two-channel greyscale, in case of alpha)
instead of being converted to three-channel sRGB. However, if a
greyscale image has a color profile, then we have to convert to sRGB,
otherwise the colors would be wrong when we strip the profile.
We also have to set the import profile, otherwise images with broken
embedded color profiles won't have a fallback profile and may get
incorrect colors. In this case we also have to be careful, because we
can't specify an sRGB fallback for greyscale or CMYK images.
Upgrade bootsnap to 1.9.3 too because Ruby 3.0.3 has a bug that causes
Rails to fail to boot when bootsnap is enabled. Bootsnap 1.9.3 works
around this bug.
Also add libgmp to build with bignum support.
Fix an issue where if you tried to remove the same vote multiple times,
nothing would happen.
Specifically, if you upvoted a post in one tab, then opened it in a
second tab, then removed your vote in the first tab, then upvoted it
again in the first tab, then tried to remove your vote again in the
second tab, nothing would happen when removing it in the second tab.
This was because we were removing votes by vote ID, which meant that if
a vote had already been removed, removing it again would do nothing
since it was already deleted.
Fix a division by zero when calculating the upvote ratio in the votes tooltip.
This could happen if the post had votes according to the post_votes
table, but didn't have any votes according to the up_score and
down_score. This should never happen, yet it did happen for post 2248039
and post 2959269.
Make the `order:random` metatag truly randomize the search. Add a
`random:N` metatag that returns up to N random posts, like what
`order:random` did before.
`order:random` now returns the entire search in random order. Before it
just returned a pageful of pseudorandom posts. This will be more
accurate for small searches, but slower for large searches. If
`order:random` times out, try `random:N` instead.
The `random:N` metatag returns up to N pseudorandom posts. This is
faster than `order:random` for large searches, but for small searches,
it may return less than N posts, and the randomness may be biased. Some
posts may be more likely than others to appear. N must be between 0 and
200.
Also, `/posts?tags=touhou&random=1` now redirects to `/posts?tags=touhou+random:N`.
Before the `random=1` param acted like a free `order:random` tag; now it
redirects to a `random:N` search, so it counts against your tag limit.
Optimize metatag searches involving usernames, including user:,
approver:, appealer:, commenter:, upvoter:, etc.
Do `User.find_by_name` instead of `User.name_matches` because this
fetches the user upfront instead of doing it inside a subquery. Using a
subquery makes the SQL more complicated and leads to worse query plans.
This especially helps searches involving multiple username metatags.
evazion
evazion