Set the MALLOC_CONF environment variable in the Docker image to tune the
Jemalloc configuration. Configuring Jemalloc to use two memory arenas
reduces memory fragmentation, and using background threads and low decay
times allows freed memory to be returned to the OS sooner.
Previously we set this environment variable at runtime in Kubernetes,
but baking it into the image is simpler.
* Add data-is-deleted, data-updater-id, data-topic-id, and
data-topic-is-deleted data attributes to forum posts.
* Change data-forum-post-id to data-id.
* Dim deleted forum posts like comments.
Add a `z=N` param to the URL when doing a search from the search box, or
when clicking a link in the tag list. This is to gather data on how
often the links in the tag list are used, and whether it's more common
to click tags in the tag list or to perform searches manually.
This is temporary and will be removed when enough data is collected.
Don't default the "Final source" field to the current source. This way
the default will be to set the source to the normalized replacement URL.
The "final source" field can be set to override this.
Fix the /posts index controller not logging the normalized search query
to NewRelic when the search failed, either because of a tag limit error,
a search timeout, or a RSS feed rate limit error.
Also don't log the number of search results when it's an API request or
failed search. This is to avoid doing a potentially slow full post count
when it's not otherwise needed.
When a banned artist tag is aliased into a nonbanned artist tag, move
the is_banned flag from the old artist entry to the new artist_entry.
Related to #4940. Fixes a case where a banned artist could lose the
banned status when it was moved.
Add foreign key constraints on all foreign keys on all tables.
These constraints are deferrable so that they're checked at the end of
the transaction, rather at the end of the statement. This is to reduce
lock duration and to allow for cyclic relationships.
Constraints are added in one migration then validated in another so that
the entire table isn't locked against reads and writes while the foreign
key constraints are being validated.
A few tables had invalid foreign keys. Add a fix script to fix these tables:
* A couple artist versions belonged to deleted artists.
* One dmail belonged to a deleted user.
* One forum topic visit belonged to that same deleted user.
* A few dozen note versions belonged to nonexistent posts. This came
from RaisingK moving notes to different posts years ago, back when it
was possible for users to set a note's post ID in the API.
* Some uploads had their parent ID set to 0.
Delete favorites that have an invalid post_id because they belong to an
expunged post.
This bug of not deleting favorites after a post is expunged was fixed
long ago, but old favorites were never cleaned up.
Fixes#4711: Some users have incorrect fav count.
Update the minimum browser requirement from browsers released since
2012, to browsers released in the last 5 years.
This means PostCSS won't prefix CSS properties like `flex` or other
properties supported by browsers since 2017. Babel also won't transpile
Javascript syntax that has been supported by browsers since 2017.
Remove a PostCSS plugin used to workaround bugs in the `flex` CSS
property. Most of these bugs are no longer relevant and only affected
ancient browsers like IE 10 or 11.
https://github.com/philipwalton/flexbugs
Don't include fallback CSS properties for CSS variables. Before we
generated CSS like this:
a.tag-type-1 {
color: #c00004;
color: var(--artist-tag-color);
}
Now we generate CSS like this:
a.tag-type-1 {
color: var(--artist-tag-color);
}
This means that support for CSS variables is now required for colors to
work properly. All major browsers have supported CSS variables since
2016-2017.
Disable the ability to click and drag thumbnails. This usually happened
by accident when you clicked on a thumbnail and accidentally dragged it,
especially during tag scripting.
Fix an open redirect exploit where if you went to <https://danbooru.donmai.us/login?url=//fakebooru.com>,
then after you logged in you would be redirected to https://fakebooru.com.
This was actually fixed by the upgrade to Rails 7.0. `redirect_to` now
raises an `UnsafeRedirectError` on redirect to an offsite URL. Before we
tried to prevent offsite redirects by checking that the URL started with
a slash, but this was insufficient - it allowed protocol-relative URLs
like `//fakebooru.com`.
Add a test case for protocol-relative URLs and return a 403 error on an
offsite redirect.
This returns a Server-Timing header on all HTTP responses, which
includes details on how long it took the server to render the response.
Browsers can show this timing information in the devtools. In Chrome, go
to the Network panel, then click a HTTP request, then click the Timing tab.
* Update framework files with `bin/rails app:update`.
* Update to use new Rails 7.0 default settings, except for a couple
things regarding new cookie and cache formats that would prevent us
from rolling back to Rails 6.1 if necessary.
Fix the paginator not detecting the first or last page correctly during
sequential pagination.
Caused by the fact that we fetch one more record than needed to detect
whether we're on the last page, then throw that record away by
overriding Rails' internal `records` method. An upstream refactoring
meant that the `size` method now counts the number of records *after*
the extra record is thrown away, where before it counted *before* the
extra record was thrown away.
Previously we overrode `ActiveRecord::Base#columns` to make it ignore
tsvector columns so they wouldn't show up in API responses. These
columns have been removed, so this is no longer necessary.
is_note_locked, is_rating_locked, and is_status_locked have been unused
since 126046cb6.
tag_index has been unused since 37a8dc5db.
fav_string has been unused since 165339236.
pool_string has been unused since 7d503f088.
Fix a bug where /jobs?search[name]=Prune+Posts didn't find jobs named PrunePostsJob.
`"Prune Posts".tr(" ", "_").classify` was wrong because it returned `"PrunePost"`.
Fix an `ActionView::Template::Error: undefined method 'verification_key'
for nil` error in the welcome_user mailer when a user signs up without
an email address.
Caused by the fact that we now render mail templates regardless of
whether the user has an email address, and then skip sending the email
only after the mail template is rendered.
Fix the /jobs listing to show retried jobs. GoodJob::ActiveJobJob has a
default scope that filters out retried jobs; remove it in the index
controller so we can see retried jobs.
Also fix mail delivery jobs not showing up in the dropdown in the /jobs
search form.
Add ability to search jobs on the /jobs page by job type or by status.
Fixes#2577 (Search filters for delayed jobs). This wasn't possible
before with DelayedJobs because it stored the job data in a YAML string,
which made it difficult to search jobs by type. GoodJobs stores job data
in a JSON object, which is easier to search in Postgres.
Remove the DelayedJobs gem and database table. Completes the transition
to GoodJob started in c06bfa64f and f4953549a.
Downstream users can upgrade as follows:
* Stop the Rails server.
* Stop the DelayedJobs worker (normally running as `bin/delayed_job` or `bin/rails jobs:work`).
* Run `bin/rails jobs:work` to finish any pending delayed jobs.
* Run `bin/rails db:migrate` to create the good_jobs table and drop the delayed_jobs table.
* Start the Rails server again.
* Start the GoodJobs worker with `bin/good_job start`.
Switch the ActiveJob backend from DelayedJob to GoodJob. Differences:
* The job worker is run with `bin/good_job start` instead of `bin/delayed_job`.
* Jobs have an 8 hour timeout instead of a 4 hour timeout.
* Jobs don't automatically retry on failure.
* Finishing jobs are preserved and pruned after 7 days.
This is the first step towards replacing DelayedJob with GoodJob. Compared to
DelayedJob:
* GoodJob supports Rails 7 (DelayedJob is currently a blocker for Rails 7
because it has a version bound on ActiveRecord <6.2).
* GoodJob has a builtin admin dashboard.
* GoodJob supports threaded job workers.
* GoodJob supports scheduled cronjobs.
* GoodJob supports healthchecks for workers.
* GoodJob uses Postgres notifications instead of polling to pick up new
jobs. This allows jobs to be picked up faster and scales better with
large numbers of workers.
https://github.com/bensheldon/good_job
Fix an `undefined method 'find' for Maintenance::User:Module` exception
when disabling email notifications using the "Disable notifications"
link in dmails.
Add a fix script to fix email addresses that are invalid or that contain
common typos. If the email can't be fixed, usually because the fixed
address is already in use by another account, then the email address is
marked undeliverable.
Fix mailers to not attempt deliveries to invalid or nonexistent email
addresses. This usually happened when someone changed their email, and
we tried to send a confirmation email to a nonexistent address.
Fix an exception when checking out on Stripe caused by using the
`allow_promotion_codes` and `discounts` parameters at the same time.
Stripe::InvalidRequestError - You may only specify one of these parameters: allow_promotion_codes, discounts
Switch the Ruby memory allocator from Glibc malloc to Jemalloc. Jemalloc
supposedly uses less memory than Glibc malloc because it's better at
handling memory fragmentation. It also has detailed internal statistics
to help monitor allocator behavior.
We use the LD_PRELOAD method of loading Jemalloc instead of building it
into Ruby so that we can switch allocators at runtime.
Further micro-optimize thumbnails to reduce string allocations.
`Post#levelblocked?` gets called once per thumbnail. Before it split the
tag string, which meant one string allocation for each tag on each post.
This added up to thousands of string allocations per pageload.
Do a few micro-optimizations to reduce the number of memory allocations
during thumbnail generation.
This commit, combined with freezing string literals in a7dc05 and
67b961, reduces the number of allocations on the front page from 180,000
to 150,000, and the number of retained objects from 8,000 to 4,000.
Monkey-patch Symbol#to_s to return a frozen (immutable) string instead
of a mutable string.
This should reduce string allocations, and thereby reduce memory usage
and garbage collector pressure, but it may be incompatible with
libraries that expect Symbol#to_s to return a mutable string.
https://bugs.ruby-lang.org/issues/16150https://github.com/Shopify/symbol-fstring
* Mark /comments/:id links in comment timestamps as nofollow to prevent
Googlebot from crawling these links.
* Mark /posts/:id/show_seq links as disallowed in robots.txt to prevent
Googlebot from crawling forward/back links on posts.
* Increase the default thumbnail size from small (150x150) to medium (180x180).
* Change the mobile layout to use three posts per row instead of two for small thumbnails.
Parent/child posts are still 150x150 to avoid taking up even more space above posts.
For small thumbnails, use 180x180 thumbnails scaled down to 150x150.
This is so we can get rid of 150x150 images and just use 180x180 for
both small and medium size thumbnails.
Also fix RSS feeds, XML sitemaps, and Discord embeds to use 360x360
thumbnails instead of 150x150 thumbnails.
* Bottom-align thumbnails on desktop. This is so that vote buttons line
up beneath thumbnails. Also so that pool titles line up on the pool
gallery page. This is desktop only because it tends to leave large
gaps above thumbnails on mobile.
* Make thumbnails fixed-height so that the layout doesn't shift as
thumbnails are loaded.
Fix the related tags section being completely hidden when it was
collapsed. The `.hidden` CSS class that was used by related tags
conflicted with the `.hidden` utility class added in 8841de68ac.
evazion
evazion