Caused by a bug in libvips-8.13.2 that was fixed in libvips issue 3027.
See libvips issue 3129 for details.
The fix is to build libvips from master until 8.13.3 is released.
Fix StatementInvalid exception when uploading https://files.catbox.moe/vxoe2p.mp4.
This was a result of multiple bugs:
* First, generating thumbnails for the video failed. This was because
the video uses the AV1 codec, which FFmpeg failed to decode. It failed
because our version of FFmpeg was built without the `--enable-libdav1d`
flag, so it uses the builtin AV1 decoder, which apparently can't
handle this particular video (it spews a bunch of errors about "Failed
to get pixel format" and "missing sequence header" and "failed to get
reference frame").
* Because generating the thumbnails failed, an exception was raised. We
tried to save the error message in the upload_media_assets.error
field. However, this also failed because the error message was 77kb
long (it contained the entire output of the ffmpeg command), but the
`upload_media_assets` table had a btree index on the `error` column,
which meant the maximum length of the error column was limited to
~2.7kb. This lead to a StatementInvalid exception being raised.
* Because the StatementInvalid exception was raised while we were trying
to set the upload media asset's status to `failed`, the upload was
left stuck in the `processing` state rather than being set to the
`failed` state.
* Because the upload was stuck in the `processing` state, the upload
page would hang forever waiting for the upload to complete.
The fixes are to:
* Build FFmpeg with `--enable-libdav1d` to use libdav1d for decoding AV1
videos instead of the builtin AV1 decoder.
* Remove the index on the `upload_media_assets.error` column so that
setting overly long error messages won't fail.
* Catch unexpected exceptions in ProcessUploadMediaAssetJob so we can
mark uploads as failed, even if `process_upload!` itself fails because
it raises an unexpected exception inside its own exception handler.
* Check that the video is playable with `MediaFile::Video#is_corrupt?` before
allowing it to be uploaded. This way we can return a better error
message if we can't generate thumbnails because the video isn't
playable. This requires decoding the entire video, so it means uploads
may take several seconds longer for long videos. It's also a security
risk in case ffmpeg has any bugs.
* Define `MediaAsset#preview!` as raising an exception on error, so
it's clear that generating thumbnails can fail. Define `MediaAsset#preview`
as returning nil on error for when we don't care about the cause of
the error.
* Update base Ubuntu image from 22.04 to 22.10.
* Update FFmpeg from 4.4.1 to 5.1.2
* Update MozJPEG from 4.0.3 to 4.1.1.
* Update ExifTool from 12.30 to 12.42.
The FFmpeg upgrade is necessary for .avif file support. Older versions
of ffprobe don't handle .avif files correctly. It either returns an
error, or in certain cases it can interpret a static .avif image as a
video (since .avif files are basically .mp4 files in disguise).
* Have CI build Docker images for both x86 and ARM.
* Add a `bin/rails danbooru:docker:build-arm` command for building a Docker image locally for ARM.
Usage:
* Test the image:
docker run --rm -it --platform linux/arm64 ghcr.io/danbooru/danbooru bash
* Build the image:
bin/rails danbooru:docker:build-arm
* Build the image by hand:
git archive HEAD | docker buildx build - --platform linux/amd64 --build-arg SOURCE_COMMIT=$(git rev-parse HEAD) -t danbooru -f Dockerfile --load
The Danbooru image now requires at least Docker 20.10.10 to run. If you
get weird errors, check `docker version` and make sure you're running a
recent enough version of Docker.
This is because Ubuntu 22.04 uses Glibc 2.34, which uses the clone3
syscall, which was blocked by Docker's default seccomp policy up until
20.10.10 [1].
You may have to upgrade your distro or install Docker manually [2] if
your distro doesn't ship a recent enough version of Docker.
A workaround for older versions of Docker is to use the
`--security-opt seccomp=unconfined` option to disable seccomp [3].
[1] https://pascalroeleven.nl/2021/09/09/ubuntu-21-10-and-fedora-35-in-docker/
[2] https://docs.docker.com/engine/install/
[3] https://docs.docker.com/engine/security/seccomp/
Add `foreman` to the base Docker image. This way you can do this:
docker run --rm -it -v $PWD:/danbooru ghcr.io/danbooru/danbooru foreman start
to start everything needed to run Danbooru in development mode (except
for the Postgres database). This will start everything listed in the
Procfile:
bin/rails server
bin/good_job start
bin/rails danbooru:cron
bin/webpack-dev-server
Set the MALLOC_CONF environment variable in the Docker image to tune the
Jemalloc configuration. Configuring Jemalloc to use two memory arenas
reduces memory fragmentation, and using background threads and low decay
times allows freed memory to be returned to the OS sooner.
Previously we set this environment variable at runtime in Kubernetes,
but baking it into the image is simpler.
Switch the Ruby memory allocator from Glibc malloc to Jemalloc. Jemalloc
supposedly uses less memory than Glibc malloc because it's better at
handling memory fragmentation. It also has detailed internal statistics
to help monitor allocator behavior.
We use the LD_PRELOAD method of loading Jemalloc instead of building it
into Ruby so that we can switch allocators at runtime.
Upgrade bootsnap to 1.9.3 too because Ruby 3.0.3 has a bug that causes
Rails to fail to boot when bootsnap is enabled. Bootsnap 1.9.3 works
around this bug.
Also add libgmp to build with bignum support.
Add a Ruby wrapper library around the libseccomp library. Seccomp is
used to restrict the syscalls a program can make. See comments in
app/logical/seccomp.rb for further details.
This is not used for anything yet. It's simply adding part of the
sandboxing infrastructure for later use.
Restructure the Dockerfile and the CSS/JS files so that we only rebuild
the CSS and JS when they change, not on every commit.
Before it took several minutes to rebuild the Docker image after every
commit, even when the JS/CSS files didn't change. This also made pulling
images slower.
This requires refactoring the CSS and JS to not use embedded Ruby (ERB)
templates, since this made the CSS and JS dependent on the Ruby
codebase, which is why we had to rebuild the assets after every Ruby
change.
Update the Postgres client binaries (psql et al) to version 14.0. This
is so they match the server version, and so that pg_amcheck is
available, which was introduced in 14.0.
This requires updating the base image to Ubuntu 21.04 at the same time
because the Postgres repo doesn't support version 14.0 on Ubuntu 20.10.
Add `less` to the Docker image to fix an issue with running `bin/rails console`.
The console uses Pry[1], which has an issue where it pipes long output
through `less`, but it tries to use the -X option, which is only
supported by GNU less, not Busybox less. There's a open bug about this
in the Pry repo dating back to 2014[2].
Add `tini` and use it as the Docker entrypoint to ensure we forward
signals to child processes and reap zombie children properly. This fixes
an issue where if you ran something like:
docker run ghcr.io/danbooru/danbooru bash -c 'bin/rails db:test:prepare && bin/rails test'
Then you couldn't use control-C to stop the container. This was because
bash wasn't forwarding signals to its children, and because by default,
programs running as PID 1 ignore SIGINT and SIGTERM. See [3][4] for details.
1: https://github.com/pry/pry
2: https://github.com/pry/pry/issues.1248
3: https://github.com/krallin/tini/issues.8
4: https://gist.github.com/StevenACoffman/41fee08e8782b411a4a26b9700ad7af5#dont-run-pid-1
Fix Exiftool not being able to get the metadata for compressed SWF
files. Exiftool requires Compress::Zlib as an optional dependency to
decompress compressed SWF files, but it wasn't in the Docker image.
Archive::Zip is required for Zip files and Digest::MD5 for certain other
metadata (see "DEPENDENCIES" in exiftool README).
Include OpenResty in the base Docker image. This is so we can run
OpenResty in front of Danbooru as a reverse proxy to serve static assets
(CSS, JS, and static images living in public/images).
Including the proxy in the same container as the static assets avoids a
lot of problems with trying to share files across separate containers.
Add a model for storing image and video metadata for uploaded files.
Metadata is extracted using ExifTool. You will need to install ExifTool
after this commit. ExifTool 12.22 is the minimum required version
because we use the `--binary` option, which was added in this release.
The MediaMetadata model is separate from the MediaAsset model because
some files contain tons of metadata, and most of it is non-essential.
The MediaAsset model represents an uploaded file and contains essential
metadata, like the file's size and type, while the MediaMetadata model
represents all the other non-essential metadata associated with a file.
Metadata is stored as a JSON column in the database.
ExifTool returns all the file's metadata, not just the EXIF metadata.
EXIF is one of several types of image metadata, hence why we call
it MediaMetadata instead of EXIFMetadata.
Make nokogiri use the bundled version of libxml2 instead of the system
version. In the past installing nokogiri was slow because it had to
compile the bundled version of libxml2, which is partly why we switched
to the system library. Now it's faster because the bundled version comes
pre-compiled with the nokogiri gem.
https://nokogiri.org/#native-gems-faster-more-reliable-installation
Reverts 440bbbb28.
* Add README files to several directories in app/ giving a brief
overview of some parts of Danbooru's architecture.
* Add documentation for files in config/.
Use the prebuilt Docker images instead of building them locally in the
Docker Compose script. This is faster, but it means that local changes
to the code will be ignored.
Fix the quickstart command in the README not working for Ubuntu 18.04.
This was because the Docker Compose file was set to version 3.7, but
Ubuntu 18.04 ships an older version of Docker Compose that only supports
version 3.4.
Fix the ca-certificates package not being installed inside the base
Docker image. This caused uploads from HTTPS sites to fail because TLS
certificates couldn't be validated.
* Optimize Dockerfile to minimize size of the Docker image.
* Specify exact versions of important dependencies (Ruby, Node, Vips) to
ensure our dependencies are up to date and locked to known versions.
* Install Vips from source because the version that ships with Ubuntu is too old.
* Install FFmpeg from source because otherwise using the Ubuntu package
pulls in tons of video libraries we don't need, bloating the image.
Add a Docker Compose file that launches a minimal Danbooru instance in a
Docker container with a single command. This is suitable as a quick demo
or for personal use, not for public-facing sites.
To use it, just run `bin/danbooru`. This is a wrapper script that
installs Docker Compose then uses it to start Danbooru.
This will generate a lot of debug output and take several minutes while
it builds the Docker containers. Be patient. When it's done, you should
have an empty booru accessible at http://localhost.
Automatically generate a random secret key for `Danbooru.config.secret_key_base`
if no key is specified.
This so that you can run Danbooru in a Docker container with zero
configuration.
This removes support for the ~/.danbooru/secret_token file and the
SECRET_TOKEN environment variable. If you used either one of these, you
must copy the value either to DANBOORU_SECRET_KEY_BASE in .env.local, or to
`secret_key_base` in config/danbooru_local_config.rb.
# .env.local
DANBOORU_SECRET_KEY_BASE=<value>
# config/danbooru_local_config.rb
def secret_key_base
# <value>
end
Replace the Google map on the IP address show page with a Bing map. Bing
doesn't require an API key, which makes it easier to deploy. The Google
Maps API requires to you to whitelist the IP addresses and domains you
plan to use with your API key, which is inconvenient for development
because it means maps won't display unless you whitelist your
development IPs.
Fix tests not working in Github. They were failing because the latest
version of Webpack needs a version of Node newer than the version in
shipped Ubuntu 20.04.
Also fix the Docker build failing because of the system timezone
database not being installed in Ubuntu 20.10.
