Add a Ruby wrapper library around the libseccomp library. Seccomp is
used to restrict the syscalls a program can make. See comments in
app/logical/seccomp.rb for further details.
This is not used for anything yet. It's simply adding part of the
sandboxing infrastructure for later use.
Restructure the Dockerfile and the CSS/JS files so that we only rebuild
the CSS and JS when they change, not on every commit.
Before it took several minutes to rebuild the Docker image after every
commit, even when the JS/CSS files didn't change. This also made pulling
images slower.
This requires refactoring the CSS and JS to not use embedded Ruby (ERB)
templates, since this made the CSS and JS dependent on the Ruby
codebase, which is why we had to rebuild the assets after every Ruby
change.
Update the Postgres client binaries (psql et al) to version 14.0. This
is so they match the server version, and so that pg_amcheck is
available, which was introduced in 14.0.
This requires updating the base image to Ubuntu 21.04 at the same time
because the Postgres repo doesn't support version 14.0 on Ubuntu 20.10.
Add `less` to the Docker image to fix an issue with running `bin/rails console`.
The console uses Pry[1], which has an issue where it pipes long output
through `less`, but it tries to use the -X option, which is only
supported by GNU less, not Busybox less. There's a open bug about this
in the Pry repo dating back to 2014[2].
Add `tini` and use it as the Docker entrypoint to ensure we forward
signals to child processes and reap zombie children properly. This fixes
an issue where if you ran something like:
docker run ghcr.io/danbooru/danbooru bash -c 'bin/rails db:test:prepare && bin/rails test'
Then you couldn't use control-C to stop the container. This was because
bash wasn't forwarding signals to its children, and because by default,
programs running as PID 1 ignore SIGINT and SIGTERM. See [3][4] for details.
1: https://github.com/pry/pry
2: https://github.com/pry/pry/issues.1248
3: https://github.com/krallin/tini/issues.8
4: https://gist.github.com/StevenACoffman/41fee08e8782b411a4a26b9700ad7af5#dont-run-pid-1
Fix Exiftool not being able to get the metadata for compressed SWF
files. Exiftool requires Compress::Zlib as an optional dependency to
decompress compressed SWF files, but it wasn't in the Docker image.
Archive::Zip is required for Zip files and Digest::MD5 for certain other
metadata (see "DEPENDENCIES" in exiftool README).
Include OpenResty in the base Docker image. This is so we can run
OpenResty in front of Danbooru as a reverse proxy to serve static assets
(CSS, JS, and static images living in public/images).
Including the proxy in the same container as the static assets avoids a
lot of problems with trying to share files across separate containers.
Add a model for storing image and video metadata for uploaded files.
Metadata is extracted using ExifTool. You will need to install ExifTool
after this commit. ExifTool 12.22 is the minimum required version
because we use the `--binary` option, which was added in this release.
The MediaMetadata model is separate from the MediaAsset model because
some files contain tons of metadata, and most of it is non-essential.
The MediaAsset model represents an uploaded file and contains essential
metadata, like the file's size and type, while the MediaMetadata model
represents all the other non-essential metadata associated with a file.
Metadata is stored as a JSON column in the database.
ExifTool returns all the file's metadata, not just the EXIF metadata.
EXIF is one of several types of image metadata, hence why we call
it MediaMetadata instead of EXIFMetadata.
Make nokogiri use the bundled version of libxml2 instead of the system
version. In the past installing nokogiri was slow because it had to
compile the bundled version of libxml2, which is partly why we switched
to the system library. Now it's faster because the bundled version comes
pre-compiled with the nokogiri gem.
https://nokogiri.org/#native-gems-faster-more-reliable-installation
Reverts 440bbbb28.
* Add README files to several directories in app/ giving a brief
overview of some parts of Danbooru's architecture.
* Add documentation for files in config/.
Use the prebuilt Docker images instead of building them locally in the
Docker Compose script. This is faster, but it means that local changes
to the code will be ignored.
Fix the quickstart command in the README not working for Ubuntu 18.04.
This was because the Docker Compose file was set to version 3.7, but
Ubuntu 18.04 ships an older version of Docker Compose that only supports
version 3.4.
Fix the ca-certificates package not being installed inside the base
Docker image. This caused uploads from HTTPS sites to fail because TLS
certificates couldn't be validated.
* Optimize Dockerfile to minimize size of the Docker image.
* Specify exact versions of important dependencies (Ruby, Node, Vips) to
ensure our dependencies are up to date and locked to known versions.
* Install Vips from source because the version that ships with Ubuntu is too old.
* Install FFmpeg from source because otherwise using the Ubuntu package
pulls in tons of video libraries we don't need, bloating the image.
Add a Docker Compose file that launches a minimal Danbooru instance in a
Docker container with a single command. This is suitable as a quick demo
or for personal use, not for public-facing sites.
To use it, just run `bin/danbooru`. This is a wrapper script that
installs Docker Compose then uses it to start Danbooru.
This will generate a lot of debug output and take several minutes while
it builds the Docker containers. Be patient. When it's done, you should
have an empty booru accessible at http://localhost.
Automatically generate a random secret key for `Danbooru.config.secret_key_base`
if no key is specified.
This so that you can run Danbooru in a Docker container with zero
configuration.
This removes support for the ~/.danbooru/secret_token file and the
SECRET_TOKEN environment variable. If you used either one of these, you
must copy the value either to DANBOORU_SECRET_KEY_BASE in .env.local, or to
`secret_key_base` in config/danbooru_local_config.rb.
# .env.local
DANBOORU_SECRET_KEY_BASE=<value>
# config/danbooru_local_config.rb
def secret_key_base
# <value>
end
Replace the Google map on the IP address show page with a Bing map. Bing
doesn't require an API key, which makes it easier to deploy. The Google
Maps API requires to you to whitelist the IP addresses and domains you
plan to use with your API key, which is inconvenient for development
because it means maps won't display unless you whitelist your
development IPs.
Fix tests not working in Github. They were failing because the latest
version of Webpack needs a version of Node newer than the version in
shipped Ubuntu 20.04.
Also fix the Docker build failing because of the system timezone
database not being installed in Ubuntu 20.10.
Fix gem version conflicts described in 20abd8a5f. Nokogiri couldn't be
upgraded past 1.10.9 because 1.11.0 causes a build failure in Nokogumbo
2.0.2, but we couldn't stay on 1.10.9 either because it has a hard
requirement on Ruby <2.7 and we require Ruby >=2.7. This made `bundle
update` fail with a Gemfile conflict.
The fix is to disable libxml2 support when building Nokogumbo. Nokogumbo
wants to use the same version of libxml2 as Nokogiri, but Nokogiri
1.11.0 changed how it reports which version of libxml2 it's using, which
causes Nokogumbo's build to fail. Disabling libxml2 may reduce
performance of Nokogumbo ([1]).
While we're at it, we also make Nokogiri use the system version of
libxml2 instead of its own bundled version. Nokogiri really wants
us to use its own patched version of libxml2 instead of the system
version, but the patches it applies look relatively minor and don't seem
relevant to us ([2]). Using the system version reduces build time during CI.
This adds libxml2 and libxslt as OS-level dependencies of Danbooru. You
may need to do `sudo apt-get install libxml2-dev libxslt-dev` to install
these libraries after this commit.
[1]: https://github.com/rubys/nokogumbo#flavors-of-nokogumbo
[2]: https://github.com/sparklemotion/nokogiri/tree/master/patches/libxml2
Store the app in /home/danbooru/app instead of in /app so that we have
permission to write inside the app dir. /app was owned by root, which
prevented the danbooru user from writing to it.
https://danbooru.donmai.us/forum_topics/9127?page=283#forum_post_160508
There was a recent outage that was caused by the read replica
(yukinoshita.donmai.us) being temporarily unavailable. The pg driver in
rails got hardstuck trying to connect to the replica, which brought down
the whole site. The app servers stopped responding and could only be
brought down with SIGKILL. Even try to boot the rails console didn't
work.
We only really used this to calculate tag counts inside Post.fast_count,
which wasn't really beneficial since the read replica is slower than the
main database.
Use git clean to delete stale files from an earlier checkout. These
files interfered with various things (specifically, a comment vote test
failed because there was an old comment_votes/create.json.erb template
hanging around that changed the behavior of the POST /comment_votes endpoint).
Upgrade procedure:
$ echo 2.6.3 > .ruby-version
# upgrade ruby-build to know about the latest ruby versions.
# see also: github.com/rkh/rbenv-update
$ cd ~/.rbenv/plugins/ruby-build && git pull
# install the version of ruby specified in .ruby-version.
# see also: github.com/capistrano/rbenv/issues/83
$ rbenv install --skip-existing
# update the shell to use the latest version of ruby by default.
$ rbenv global 2.6.3
# update rubygems and bundler
$ gem update --system
# update the lockfile to use the latest version of bundler
$ bundle update --bundler
