This is the first step towards replacing DelayedJob with GoodJob. Compared to
DelayedJob:
* GoodJob supports Rails 7 (DelayedJob is currently a blocker for Rails 7
because it has a version bound on ActiveRecord <6.2).
* GoodJob has a builtin admin dashboard.
* GoodJob supports threaded job workers.
* GoodJob supports scheduled cronjobs.
* GoodJob supports healthchecks for workers.
* GoodJob uses Postgres notifications instead of polling to pick up new
jobs. This allows jobs to be picked up faster and scales better with
large numbers of workers.
https://github.com/bensheldon/good_job
Remove the SFTP file storage backend. Downstream users can use either
sshfs (which is what Danbooru now uses in production) or rclone instead.
The Ruby SFTP gem was much slower than sshfs.
Add a Ruby wrapper library around the libseccomp library. Seccomp is
used to restrict the syscalls a program can make. See comments in
app/logical/seccomp.rb for further details.
This is not used for anything yet. It's simply adding part of the
sandboxing infrastructure for later use.
Only load the `listen` and `solargraph` gems in the development
environment, not the test environment. The `listen` gem automatically
spawns background threads to listen for file changes, in order to
automatically reload code when files change, which we don't want or need
in test mode. These threads can interfere with sandboxing, because they
prevent us from being able to call unshare(2) (which can only be called
from a single-threaded process).
No longer used now that we use Puma in production. If you still used
Unicorn in your install, switch to `bin/rails server` instead. See
config/puma.rb for config settings.
No longer used now that we use Kubernetes to deploy the site instead of
Capistrano.
If you run your own installation of Danbooru, and you used Capistrano to
deploy your site, it is recommended that you switch to either the Docker
Compose file (for personal installs), the Procfile (for non-Dockerized,
development environments), or Kubernetes (for production environments;
see https://github.com/danbooru/danbooru-infrastructure/tree/master/k8s
for Danbooru's production configuration).
When processing an alias, rename, implication, mass update, or nuke,
update the posts in parallel. This means that if we alias foo to bar,
for example, then we use four processes at once to retag the posts from
foo to bar.
This doesn't mean that if we have two aliases in a BUR, we process both
aliases in parallel. It simply means that when processing an alias, we
update the posts in parallel for that alias.
Unlike Unicorn, Puma doesn't have a builtin HTTP request timeout
mechanism, so we have to use Rack::Timeout instead.
See the caveats in the Rack::Timeout documentation [1]. In Unicorn, a
timeout would send a SIGKILL to the worker, immediately killing it. This
would result in a dropped connection and a Cloudflare 502 error to the
user. In Puma, it raises an exception, which we can catch and return a
better error to the user. On the other hand, raising an exception can
potentially corrupt application state if it's sent at the wrong time, or
be delayed indefinitely if the app is stuck in IO or C extension code.
The default request timeout is 65 seconds. 65 seconds is to give things
like HTTP requests on a 60 second timeout enough time to complete. Set
the RACK_REQUEST_TIMEOUT environment variable to change the timeout.
1: https://github.com/sharpstone/rack-timeout#further-documentation
Fix an issue where the New Relic agent always started in the production
environment, even when a license key wasn't configured.
Also make the New Relic agent log to stdout instead of log/newrelic_agent.log.
Remove a workaround added in 2c06766c9. meta_request had a bug that
caused Rails to fail to launch under Rails 6.1. The fix was finally
merged upstream.
hxxps://github.com/dejan/rails_panel/pull/177.
* Export daily public database dumps to BigQuery and Google Cloud Storage.
* Only data visible to anonymous users is exported. Some tables have
null or missing fields because of this.
* The bans table is excluded because some bans have an expires_at
timestamp set beyond year 9999, which BigQuery doesn't support.
* The favorites table is excluded because it's too slow to dump (it
doesn't have an id index, which is needed by find_each).
* Version tables are excluded because dumping them every day is
inefficient, streaming insertions should be used instead.
Links:
* https://console.cloud.google.com/bigquery?project=danbooru1
* https://console.cloud.google.com/storage/browser/danbooru_public
* https://storage.googleapis.com/danbooru_public/data/posts.json
Add a new color palette and rework all site colors (both light mode and dark mode) to
use the new palette.
This ensures that colors are used consistently, from a carefully designed color palette,
instead of being chosen at random.
Before, colors in light mode were chosen on an ad-hoc basis, which resulted in a lot of
random colors and inconsistent design.
The new palette has 7 hues: red, orange, yellow, green, blue, azure (a lighter blue), and
purple. There's also a greyscale. Each hue has 10 shades of brightness, which (including
grey) gives us 80 total colors.
Colors are named like this:
var(--red-0); /* very light red */
var(--red-2); /* light red */
var(--red-5); /* medium red */
var(--red-7); /* dark red */
var(--red-9); /* very dark red */
var(--green-7); /* dark green */
var(--blue-5); /* medium blue */
var(--purple-3); /* light purple */
/* etc */
The color palette is designed to meet the following criteria:
* To have close equivalents to the main colors used in the old color scheme,
especially tag colors, so that changes to major colors are minimized.
* To produce a set of colors that can be used as as main text colors, as background
colors, and as accent colors, both in light mode and dark mode.
* To ensure that colors at the same brightness level have the same perceived brightness.
Green-4, blue-4, red-4, purple-4, etc should all have the same brightness and contrast
ratios. This way colors look balanced. This is actually a difficult problem, because human
color perception is non-linear, so you can't just scale brightness values linearly.
There's a color palette test page at https://danbooru.donmai/static/colors
Notable changes to colors in light mode:
* Username colors are the same as tag colors.
* Copyright tags are a deeper purple.
* Builders are a deeper purple (fixes#4626).
* Moderators are green.
* Gold users are orange.
* Parent borders are a darker green.
* Child borders are a darker orange.
* Unsaved notes have a thicker red border.
* Selected notes have a thicker blue (not green) border.
Fix tests not working in Github. They were failing because the latest
version of Webpack needs a version of Node newer than the version in
shipped Ubuntu 20.04.
Also fix the Docker build failing because of the system timezone
database not being installed in Ubuntu 20.10.
Upgrade to webpacker-6.0.0.beta.4 to fix a bug where bin/webpack-dev-server
didn't respect the host/port config options in config/webpacker.yml,
which made it listen on the wrong port, breaking hot reloading.
meta_request fails in Rails 6.1 with a `SystemStackError: stack level
too deep` error. Switch to a patched fork until the mainline gem is
fixed.
* hxxps://www.github.com/dejan/rails_panel/pull/177
* hxxps://www.github.com/dejan/rails_panel/issues/178
Refactor the post preview html to use the ViewComponent framework. This
lets us encapsulate all the HTML, CSS, and helper methods for a UI
component in a single place.
See https://viewcomponent.org.
* Swap out activerecord-hierarchical_query gem for some guy's patched
version because the mainline version is incompatible with 6.1.
* Disable meta_request gem because it hangs puma on startup on 6.1.
* Test that the user upgrade process integrates with Stripe correctly.
* Replace a deprecated `card` param with `source` in `Stripe::Charge.create`.
* Rescue Stripe::StripeError instead of Stripe::CardError so that we
handle failures outside of card failures, such as network errors.
Upgrade the http-cookie gem to a personal fork containing a bugfix for a
http-cookie bug that is triggered by Rails 6.1.
The bug is that HTTP::Cookie objects raise an exception if they're
compared against non-cookie objects. This bug gets triggered when
the Nijie source strategy calls `Rails.cache.fetch` to cache the
Nijie login cookie. `Rails.cache.fetch` ends up calling
ActiveSupport::Cache::Store::Entry#dup_value!, which compares the cookie
with `true`, which triggers the exception.
The http-cookie gem hasn't been updated for 4 years, so we're stuck
patching the library ourselves.
Comparison:
* Codecov has a simpler integration and a better UI.
* Codeclimate tracks both linter warnings (Rubocop, ESLint) and code
coverage, but its UI for code coverage is worse than Codecov's.
* Codeclimate doesn't support Simplecov 0.18 because Codeclimate doesn't
support 0.18's new coverage format yet.
Fix gem version conflicts described in 20abd8a5f. Nokogiri couldn't be
upgraded past 1.10.9 because 1.11.0 causes a build failure in Nokogumbo
2.0.2, but we couldn't stay on 1.10.9 either because it has a hard
requirement on Ruby <2.7 and we require Ruby >=2.7. This made `bundle
update` fail with a Gemfile conflict.
The fix is to disable libxml2 support when building Nokogumbo. Nokogumbo
wants to use the same version of libxml2 as Nokogiri, but Nokogiri
1.11.0 changed how it reports which version of libxml2 it's using, which
causes Nokogumbo's build to fail. Disabling libxml2 may reduce
performance of Nokogumbo ([1]).
While we're at it, we also make Nokogiri use the system version of
libxml2 instead of its own bundled version. Nokogiri really wants
us to use its own patched version of libxml2 instead of the system
version, but the patches it applies look relatively minor and don't seem
relevant to us ([2]). Using the system version reduces build time during CI.
This adds libxml2 and libxslt as OS-level dependencies of Danbooru. You
may need to do `sudo apt-get install libxml2-dev libxslt-dev` to install
these libraries after this commit.
[1]: https://github.com/rubys/nokogumbo#flavors-of-nokogumbo
[2]: https://github.com/sparklemotion/nokogiri/tree/master/patches/libxml2
