Bug: if ExifTool exited with status 1 because it thought the file was
corrupt, then we didn't record any of the metadata, even though it was
able to read most of it. It turns out there are thousands of posts with
minorly corrupt metadata that ExifTool is still able to read, but will
complain about.
Fix: ignore the exit code of ExifTool and always save whatever metadata
ExifTool is able to return. It will return an `ExifTool:Error` tag in
the event of errors.
Note that there are some (many?) files that are considered corrupt by
ExifTool but not by Vips, and vice versa. Probably because ExifTool only
parses the metadata while Vips only parses the image data.
Fix Exiftool not being able to get the metadata for compressed SWF
files. Exiftool requires Compress::Zlib as an optional dependency to
decompress compressed SWF files, but it wasn't in the Docker image.
Archive::Zip is required for Zip files and Digest::MD5 for certain other
metadata (see "DEPENDENCIES" in exiftool README).
Unlike Unicorn, Puma doesn't have a builtin HTTP request timeout
mechanism, so we have to use Rack::Timeout instead.
See the caveats in the Rack::Timeout documentation [1]. In Unicorn, a
timeout would send a SIGKILL to the worker, immediately killing it. This
would result in a dropped connection and a Cloudflare 502 error to the
user. In Puma, it raises an exception, which we can catch and return a
better error to the user. On the other hand, raising an exception can
potentially corrupt application state if it's sent at the wrong time, or
be delayed indefinitely if the app is stuck in IO or C extension code.
The default request timeout is 65 seconds. 65 seconds is to give things
like HTTP requests on a 60 second timeout enough time to complete. Set
the RACK_REQUEST_TIMEOUT environment variable to change the timeout.
1: https://github.com/sharpstone/rack-timeout#further-documentation
Include OpenResty in the base Docker image. This is so we can run
OpenResty in front of Danbooru as a reverse proxy to serve static assets
(CSS, JS, and static images living in public/images).
Including the proxy in the same container as the static assets avoids a
lot of problems with trying to share files across separate containers.
Change this message:
2 post(s) on this page were hidden by safe mode. Go to Danbooru or
disable safe mode to view them (learn more).
To link to [[help:safe mode]] instead of [[help:user settings]].
Update the config for the Puma webserver (used by `bin/rails server`).
* Update default settings.
* Prefix all Puma environment variables with `PUMA_`.
* Enable the Puma control app (`bin/pumactl`).
Fix a bug where if you did a slow search that took too long to calculate
the page count, and you had 200 posts per page, then we would show page
5000 as the last page of the search.
This was because we were artificially returning 1,000,000 as the post
count to signal that the count timed out, but at 200 posts per page this
would show 5000 as the last page of the search.
Add a model for storing image and video metadata for uploaded files.
Metadata is extracted using ExifTool. You will need to install ExifTool
after this commit. ExifTool 12.22 is the minimum required version
because we use the `--binary` option, which was added in this release.
The MediaMetadata model is separate from the MediaAsset model because
some files contain tons of metadata, and most of it is non-essential.
The MediaAsset model represents an uploaded file and contains essential
metadata, like the file's size and type, while the MediaMetadata model
represents all the other non-essential metadata associated with a file.
Metadata is stored as a JSON column in the database.
ExifTool returns all the file's metadata, not just the EXIF metadata.
EXIF is one of several types of image metadata, hence why we call
it MediaMetadata instead of EXIFMetadata.
Fix the Docker image not including the git hash of the build. On
DockerHub we had to set it with a build hook, but now with Github we
set it as part of the build-push-action.
Fix a bug where generating thumbnails failed for certain images when
using libvips 8.10. Specifically, it failed for single-channel greyscale
images and four-channel CMYK images without an embedded color profile.
In these cases we specified an sRGB fallback profile, but under libvips
8.10 this failed because the sRGB profile was incompatible with
single-channel and four-channel images. Before libvips 8.10 this worked,
but as of 8.10 it's a hard error.
The way libvips handles fallback color profiles differs across versions,
so we have to use different arguments for different versions. In 8.7,
vips doesn't have builtin color profiles, so we have to specify our own
manually. In 8.9, it has builtin profiles, so we can omit the import
profile, but we're still required to set the export profile to sRGB,
otherwise it will leave CMYK images as CMYK when generating thumbnails.
In 8.10, we have to _not_ to set the import or export profile to sRGB,
otherwise it will fail with an incompatible profile error when it tries
to convert CMYK images to RGB.
The builtin sRGB profile used by libvips[1] is different than the one we
used previously[2]. The builtin one comes from LCMS[3], whereas ours
came from ArgyllCMS.[4] Not all sRGB profiles are created the same[5],
so this may result in some imperceptible differences in thumbnail
output. The ArgyllCMS profile was used before because it seemed to be
the best one[6], but realistically it probably doesn't matter.
1: https://github.com/libvips/libvips/blob/v8.10.6/libvips/colour/profiles/sRGB.icm
2: https://github.com/danbooru/danbooru/blob/906eec190d45d58941f83f5ec8165ddeba205317/config/sRGB.icm
3: https://www.littlecms.com/
4: https://www.argyllcms.com/
5: https://ninedegreesbelow.com/photography/srgb-profile-comparison.html
6: https://ninedegreesbelow.com/photography/srgb-profile-comparison.html#addendum
Run the tests using the Docker image we just built, instead of
installing everything manually and running the tests outside of Docker.
Ensures the Docker image we built really works.
Account upgrades are now logged on the /user_upgrades page, so they
no longer need to be recorded as mod actions. The mod actions log should
be reserved for privileged actions performed by Builders and above. They
also tended to spam the mod actions log.
Hardcode the list of nondisposable email providers instead of making it
a config option. Also add a few new providers.
This was previously a config option to keep it secret, but there's not
much need for secrecy here.
A Restricted user's email must be on this list to unrestrict their
account. If a user is Restricted and their email is not in this list,
then it's assumed to be disposable and can't be used to unrestrict their
account even if they verify their email address.
Generate smart previews as .png so we don't suffer recompression losses
when we convert the preview frame from a full size image down to a
150x150 .jpg thumbnail.
Fix regression in ef2857667 that caused animated GIFs and PNGs to
generate thumbnails that were larger than 150x150.
Also fix a bug with cropped previews not being generated for animated
GIFs and PNGs.
Remove StorageManager::Hybrid and StorageManager::Match. These were used
to store uploads on different servers based on the post ID or file
sample type. This is no longer used in production because in hindsight
it's a lot more difficult to manage uploads when they're fragmented
across different servers.
If you need this, you can do tricks with network filesystems to get the
same effect. For example, if you want to store some files on server A
and others on server B, then mount servers A and B as network
filesystems (with e.g. sshfs, Samba, NFS, etc), and use symlinks to
point subdirectories at either server A or B.
Fix https://github.com/danbooru/danbooru/issues/4783#issuecomment-912426739:
Spoke too soon. It works in flash posts, but not in non-flash posts.
Here's the full trace that happens by trying the Shift-E shortcut
(tested on https://danbooru.donmai.us/posts/4749304 with the latest
ruffle for chrome at https://ruffle.rs/#downloads):
4749304:7 Uncaught TypeError: Cannot read properties of undefined (reading 'appendChild')
at Object.initialize_ruffle_player (application-237cebbe071cb34ce095.js:1)
at Object.initialize_all (application-237cebbe071cb34ce095.js:1)
at HTMLDocument.<anonymous> (application-237cebbe071cb34ce095.js:1)
at f (857-ff5115a2d6ceb23b4291.js:2)
at p (857-ff5115a2d6ceb23b4291.js:2)
at nrWrapper (4749304?q=status%3Aany:7)
Caused by trying to initialize the Ruffle player on non-Flash posts,
because we checked for the existence of the `window.RufflePlayer` object
to tell if we were on a Flash post, but when using the Ruffle browser
extension the RufflePlayer object will always exist.
Caused by Webpack clobbering the `Danbooru` variable for unknown reasons
when the app/javascript/packs/flash.js pack is loaded. Disabling the
output.library option seems to fix it.
A MediaAsset represents an image or video file uploaded to Danbooru. It
stores the metadata associated with the image or video. This is to work
on decoupling files from posts so that images can be uploaded separately
from posts.
DEPRECATION WARNING: `.reorder(nil)` with `.first` / `.first!` no
longer takes non-deterministic result in Rails 6.2. To continue
taking non-deterministic result, use `.take` / `.take!` instead.
(called from random at /home/user/src/danbooru/app/controllers/posts_controller.rb:91)
Make nokogiri use the bundled version of libxml2 instead of the system
version. In the past installing nokogiri was slow because it had to
compile the bundled version of libxml2, which is partly why we switched
to the system library. Now it's faster because the bundled version comes
pre-compiled with the nokogiri gem.
https://nokogiri.org/#native-gems-faster-more-reliable-installation
Reverts 440bbbb28.
Allow moderators to search `disapproved:<username>` with any user.
Before mods could only search for their own disapprovals, even though
they could see disapprovals by others.
Fix an exploit that let you determine the flagger of a post using
`flagger:<username>` saved searches. Saved searches were performed as
DanbooruBot, but since DanbooruBot is a moderator, it let unprivileged
users do `flagger:<username>` searches. Saved searches were done as a
moderator to avoid tag limits, but this is no longer necessary since the
last PostQueryBuilder refactor.
fred get out
Fix exception during https://danbooru.donmai.us/posts/random?tags=ordfav:nonamethanks
Before we were doing a query like this:
SELECT
"posts".*
FROM
"posts"
INNER JOIN
"favorites" ON "favorites"."post_id" = "posts"."id"
WHERE
(favorites.user_id % 100 = 64 AND favorites.user_id = 52664)
AND "posts"."id" = 343894
ORDER BY
favorites.id DESC,
posts.id DESC,
ID=343894 DESC
but `ID=? DESC` is ambiguous during an ordfav: search because of the
join on the favorites table. The fix is to qualify the reference as
`posts.id`.
Pundit 2.1.1 changed it so that if the first argument to `authorize` is
an Array, then the `authorize` call returns the last element of the
array. This broke order:random, because in that case we returned an
Array of posts. The fix is to return an ActiveRecord::Relation of posts,
which is more correct anyway.
Fix a regression introduced in rails/rails@4b1122c with the upgrade to Rails
6.1.4.1.
Triggered by a call to `SELECT * FROM ip_geolocations WHERE ip_addr in ...`.
A Rails refactoring changed the way that `WHERE ... IN ...` statements
worked, which had the side effect of passing a string value to our
IpAddressType serializer where before we expected a Danbooru::IpAddress
object.
Add support for using a proxy for HTTP requests. Only used for external
requests, such as downloading files or talking to source sites such as
Pixiv or Twitter, not for internal requests, such as talking to IQDB or
Reportbooru.
Using `Rails.logger` here causes server boot to fail with a `Undefined
method 'tagged'` error, possibly because `Rails.logger` isn't ready yet
during early initialization.
Parse the user agent and log whether it seems like a known bot or a
human to NewRelic under the `user.bot` request attribute. This is so
that known bots can be filtered out of search traffic analytics. Bots
and search crawlers make up a significant portion of search traffic.
* Add README files to several directories in app/ giving a brief
overview of some parts of Danbooru's architecture.
* Add documentation for files in config/.
Fix `PostVersionTest` being defined in two different places, which broke
the test runner if you tried to run the system tests at the same time as
the regular tests.
evazion
evazion