Unlike Unicorn, Puma doesn't have a builtin HTTP request timeout
mechanism, so we have to use Rack::Timeout instead.
See the caveats in the Rack::Timeout documentation [1]. In Unicorn, a
timeout would send a SIGKILL to the worker, immediately killing it. This
would result in a dropped connection and a Cloudflare 502 error to the
user. In Puma, it raises an exception, which we can catch and return a
better error to the user. On the other hand, raising an exception can
potentially corrupt application state if it's sent at the wrong time, or
be delayed indefinitely if the app is stuck in IO or C extension code.
The default request timeout is 65 seconds. 65 seconds is to give things
like HTTP requests on a 60 second timeout enough time to complete. Set
the RACK_REQUEST_TIMEOUT environment variable to change the timeout.
1: https://github.com/sharpstone/rack-timeout#further-documentation
Include OpenResty in the base Docker image. This is so we can run
OpenResty in front of Danbooru as a reverse proxy to serve static assets
(CSS, JS, and static images living in public/images).
Including the proxy in the same container as the static assets avoids a
lot of problems with trying to share files across separate containers.
Change this message:
2 post(s) on this page were hidden by safe mode. Go to Danbooru or
disable safe mode to view them (learn more).
To link to [[help:safe mode]] instead of [[help:user settings]].
Update the config for the Puma webserver (used by `bin/rails server`).
* Update default settings.
* Prefix all Puma environment variables with `PUMA_`.
* Enable the Puma control app (`bin/pumactl`).
Fix a bug where if you did a slow search that took too long to calculate
the page count, and you had 200 posts per page, then we would show page
5000 as the last page of the search.
This was because we were artificially returning 1,000,000 as the post
count to signal that the count timed out, but at 200 posts per page this
would show 5000 as the last page of the search.
Add a model for storing image and video metadata for uploaded files.
Metadata is extracted using ExifTool. You will need to install ExifTool
after this commit. ExifTool 12.22 is the minimum required version
because we use the `--binary` option, which was added in this release.
The MediaMetadata model is separate from the MediaAsset model because
some files contain tons of metadata, and most of it is non-essential.
The MediaAsset model represents an uploaded file and contains essential
metadata, like the file's size and type, while the MediaMetadata model
represents all the other non-essential metadata associated with a file.
Metadata is stored as a JSON column in the database.
ExifTool returns all the file's metadata, not just the EXIF metadata.
EXIF is one of several types of image metadata, hence why we call
it MediaMetadata instead of EXIFMetadata.
Fix the Docker image not including the git hash of the build. On
DockerHub we had to set it with a build hook, but now with Github we
set it as part of the build-push-action.
Fix a bug where generating thumbnails failed for certain images when
using libvips 8.10. Specifically, it failed for single-channel greyscale
images and four-channel CMYK images without an embedded color profile.
In these cases we specified an sRGB fallback profile, but under libvips
8.10 this failed because the sRGB profile was incompatible with
single-channel and four-channel images. Before libvips 8.10 this worked,
but as of 8.10 it's a hard error.
The way libvips handles fallback color profiles differs across versions,
so we have to use different arguments for different versions. In 8.7,
vips doesn't have builtin color profiles, so we have to specify our own
manually. In 8.9, it has builtin profiles, so we can omit the import
profile, but we're still required to set the export profile to sRGB,
otherwise it will leave CMYK images as CMYK when generating thumbnails.
In 8.10, we have to _not_ to set the import or export profile to sRGB,
otherwise it will fail with an incompatible profile error when it tries
to convert CMYK images to RGB.
The builtin sRGB profile used by libvips[1] is different than the one we
used previously[2]. The builtin one comes from LCMS[3], whereas ours
came from ArgyllCMS.[4] Not all sRGB profiles are created the same[5],
so this may result in some imperceptible differences in thumbnail
output. The ArgyllCMS profile was used before because it seemed to be
the best one[6], but realistically it probably doesn't matter.
1: https://github.com/libvips/libvips/blob/v8.10.6/libvips/colour/profiles/sRGB.icm
2: 906eec190d/config/sRGB.icm
3: https://www.littlecms.com/
4: https://www.argyllcms.com/
5: https://ninedegreesbelow.com/photography/srgb-profile-comparison.html
6: https://ninedegreesbelow.com/photography/srgb-profile-comparison.html#addendum
Run the tests using the Docker image we just built, instead of
installing everything manually and running the tests outside of Docker.
Ensures the Docker image we built really works.
Account upgrades are now logged on the /user_upgrades page, so they
no longer need to be recorded as mod actions. The mod actions log should
be reserved for privileged actions performed by Builders and above. They
also tended to spam the mod actions log.
Hardcode the list of nondisposable email providers instead of making it
a config option. Also add a few new providers.
This was previously a config option to keep it secret, but there's not
much need for secrecy here.
A Restricted user's email must be on this list to unrestrict their
account. If a user is Restricted and their email is not in this list,
then it's assumed to be disposable and can't be used to unrestrict their
account even if they verify their email address.
Generate smart previews as .png so we don't suffer recompression losses
when we convert the preview frame from a full size image down to a
150x150 .jpg thumbnail.
Fix regression in ef2857667 that caused animated GIFs and PNGs to
generate thumbnails that were larger than 150x150.
Also fix a bug with cropped previews not being generated for animated
GIFs and PNGs.
Remove StorageManager::Hybrid and StorageManager::Match. These were used
to store uploads on different servers based on the post ID or file
sample type. This is no longer used in production because in hindsight
it's a lot more difficult to manage uploads when they're fragmented
across different servers.
If you need this, you can do tricks with network filesystems to get the
same effect. For example, if you want to store some files on server A
and others on server B, then mount servers A and B as network
filesystems (with e.g. sshfs, Samba, NFS, etc), and use symlinks to
point subdirectories at either server A or B.
Fix https://github.com/danbooru/danbooru/issues/4783#issuecomment-912426739:
Spoke too soon. It works in flash posts, but not in non-flash posts.
Here's the full trace that happens by trying the Shift-E shortcut
(tested on https://danbooru.donmai.us/posts/4749304 with the latest
ruffle for chrome at https://ruffle.rs/#downloads):
4749304:7 Uncaught TypeError: Cannot read properties of undefined (reading 'appendChild')
at Object.initialize_ruffle_player (application-237cebbe071cb34ce095.js:1)
at Object.initialize_all (application-237cebbe071cb34ce095.js:1)
at HTMLDocument.<anonymous> (application-237cebbe071cb34ce095.js:1)
at f (857-ff5115a2d6ceb23b4291.js:2)
at p (857-ff5115a2d6ceb23b4291.js:2)
at nrWrapper (4749304?q=status%3Aany:7)
Caused by trying to initialize the Ruffle player on non-Flash posts,
because we checked for the existence of the `window.RufflePlayer` object
to tell if we were on a Flash post, but when using the Ruffle browser
extension the RufflePlayer object will always exist.
Caused by Webpack clobbering the `Danbooru` variable for unknown reasons
when the app/javascript/packs/flash.js pack is loaded. Disabling the
output.library option seems to fix it.
A MediaAsset represents an image or video file uploaded to Danbooru. It
stores the metadata associated with the image or video. This is to work
on decoupling files from posts so that images can be uploaded separately
from posts.
DEPRECATION WARNING: `.reorder(nil)` with `.first` / `.first!` no
longer takes non-deterministic result in Rails 6.2. To continue
taking non-deterministic result, use `.take` / `.take!` instead.
(called from random at /home/user/src/danbooru/app/controllers/posts_controller.rb:91)
