Allow uploading .zip, .rar, and .7z files from disk. The archive will be extracted and the images
inside will be uploaded.
This only works for archive files uploaded from disk, not from a source URL.
Post source URLs will look something like this: "file://foo.zip/1.jpg", "file://foo.zip/2.jpg", etc.
Sometimes artists uses Shift JIS or other encodings instead of UTF-8 for filenames. In these cases
we just assume the filename is UTF-8 and replace invalid characters with '?', so filenames might be
wrong in some cases.
There are various protections to prevent uploading malicious archive files:
* Archives with more than 100 files aren't allowed.
* Archives that decompress to more than 100MB aren't allowed.
* Archives with filenames containing '..' components aren't allowed (e.g. '../../../../../etc/passwd').
* Archives with filenames containing absolute paths aren't allowed (e.g. '/etc/passwd').
* Archives containing symlinks aren't allowed (e.g. 'foo -> /etc/passwd').
* Archive types other than .zip, .rar, and .7z aren't allowed (e.g. .tar.gz, .cpio).
* File permissions, owners, and other metadata are ignored.
Partial fix for #5340: Add support for extracting archive attachments from certain sources
Fix temp files generated during the upload process not being cleaned up quickly enough. This included
downloaded files, generated preview images, and Ugoira video conversions.
Before we relied on `Tempfile` cleaning up files automatically. But this only happened when the
Tempfile object was garbage collected, which could take a long time. In the meantime we could have
hundreds of megabytes of temp files hanging around.
The fix is to explicitly close temp files when we're done with them. But the standard `Tempfile`
class doesn't immediately delete the file when it's closed. So we also have to introduce a
Danbooru::Tempfile wrapper that deletes the tempfile as soon as it's closed.
Don't log a mod action when a user deletes their own account. This isn't a moderator action, so it
doesn't belong here. Account deletions are still logged on the /user_events page (visible to mods only).
A mod action is still logged when the Owner-level user deletes someone else's account.
Fix `Relation passed to #and must be structurally compatible. Incompatible values: [:joins] (ArgumentError)`
exception in `ordfav:evazion ratio:4:3` search. Broken by e849d8f1c.
We were effectively doing this:
q1 = Post.joins(:favorites, :media_asset).where("favorites.user_id = ?", 52664).order("favorites.id DESC")
q2 = Post.joins(:media_asset, :favorites).where("ROUND(media_assets.image_width::numeric / media_assets.image_height::numeric, 2) = 1.33")
q3 = q1.and(q2)
This failed because Rails didn't like the fact that the joins were in a different order when the
queries were `and`-ed together.
Don't allow users to choose names that conflict with search syntax, like `any` or `none`, or names
that impersonate user levels, like `Admin`, `Moderator`, `Anonymous`, etc.
* Don't delete the user's favorites unless private favorites are enabled. The general rule is that
public account activity is kept and private account activity is deleted.
* Delete the user's API keys, forum topics visits, private favgroups, downvotes, and upvotes (if
privacy is enabled).
* Reset all of the user's account settings to default. This means custom CSS is deleted, where it
wasn't before.
* Delete everything but the user's name and password asynchronously.
* Don't log the current user out if it's the owner deleting another user's account.
* Fix#5067 (Mod actions sometimes not created for user deletions) by wrapping the deletion process
in a transaction.
Automatically add the `sound` tag if the post has sound. Remove the tag if the post doesn't have sound.
A video is considered to have sound if its peak loudness is greater than -70 dB. The current quietest post
on Danbooru has a peak loudness of -62 dB (post #3470668), but it's possible to have audible sound at
-80 dB or possibly even lower. It's hard to draw a clear line between "silent" and "barely audible".
At first we rounded loudness values to 4 decimal places to make them
easier to compare. This meant the lowest level was 0.0001, or -80 dB,
but it's possible for volume levels to go even lower than that.
For videos with sound, save information about audio volume levels in the
media asset's metadata. These values are stored:
* FFmpeg:AudioPeakLoudness The peak loudness of the audio track, from 0.0 (silent) to 1.0 (max volume)
* FFmpeg:AudioAverageLoudness The average loudness of the audio track, from 0.0 (silent) to 1.0 (max volume).
* FFmpeg:AudioLoudnessRange The difference between the quietest and loudest sounds in the audio track (in decibels).
* FFmpeg:AudioSilencePercentage The percentage of the video that is silent (1.0 is completely silent, 0.5 is 50% silence, 0.0 is no silence).
These values are calculated based on the EBU R 128 standard, using the ffmpeg command below:
ffmpeg -i file.mp4 -af silencedetect=duration=0.05:noise=0.0001,ebur128=metadata=1:peak=true:dualmono=true -f null /dev/null
See the links below for details:
* https://en.wikipedia.org/wiki/EBU_R_128
* https://www.ffmpeg.org/ffmpeg-filters.html#ebur128-1
* https://tech.ebu.ch/loudness
* https://tech.ebu.ch/docs/tech/tech3341.pdf
Fix .webm files not including the `FFmpeg:VideoBitRate` and `FFmpeg:AudioBitRate`
fields in the media_metadata table. This was because the .webm format
doesn't include the video or audio bit rates in the metadata, and
ffprobe doesn't calculate them either, so we have to calculate them
ourselves by hand.
Fixup for 523d7afdd.
If a media asset is corrupt, include the error message from libvips or
ffmpeg in the "Vips:Error" or "FFmpeg:Error" fields in the media
metadata table.
Corrupt files can't be uploaded nowadays, but they could be in the past,
so we have some old corrupted files that we can't generate thumbnails
for. This lets us mark these files in the metadata so they're findable
with the tag search `exif:Vips:Error`.
Known bug: Vips has a single global error buffer that is shared between
threads and that isn't cleared between operations. So we can't reliably
get the actual error message because it may pick up errors from other
threads, or from previous operations in the same thread.
When searching posts by width, height, file size, or file extension, use the
values from the media_assets table rather than the posts table.
This makes filetype: searches faster because the file_ext is indexed on
the media assets table, but not on the posts table.
This paves the way for getting rid of the width, height, file_size, and
file_ext indexes on the posts table in the future. It's wasteful to
index these columns on both the posts table and the media assets table.
Use ExifTool to get the dimensions of Flash files instead of calculating
it ourselves. Avoids copying third-party code.
Fixes a bug where Flash files with fractional dimensions (e.g. 607.6 x 756.6)
had their dimensions rounded down instead of rounded up.
Fixes another bug where Flash files could return negative dimensions.
This happened for two files:
* https://danbooru.donmai.us/media_assets/228662 (-179.2 x -339.2)
* https://danbooru.donmai.us/media_assets/228664 (-179.2 x -339.2)
Now we round these up to 1x1. This is still wrong, but it's less wrong than before.
Fix certain corrupt GIFs returning dimensions of 0x0. This happened
when the GIF was too corrupt for libvips to read. Fixed by using
ExifTool to read the dimensions instead.
Also add validations to ensure that it's not possible to have media
assets with a width or height of 0.
Previously the width, height, and file type fields returned by ExifTool
weren't saved in the media metadata because they were already saved in
the media asset. However, in some cases, it can be useful to compare
ExifTool's version of these fields with our own. This can be useful when
an image is corrupt and libvips can't get the width or height, or when
it's a video and we want to make sure we detected the correct type of video.
script/files/123_refresh_media_metatadata.rb needs to be run after this
to update the metadata.
Fix a bug where MP4 files with major brand "iso4" weren't detected as
MP4, so they couldn't be uploaded.
This switches our MP4 detection code to something very similar to Firefox's
MP4 sniffing algorithm. Ours is slightly wrong because a) we only check
the major_brand, not the minor_brands, and b) we falsely detect certain 3GP
videos as MP4. 3GP is a very similar format to MP4, close enough that it
can be played by Chrome (but not Firefox), but it's technically not MP4
and should not have a .mp4 file extension. We leave it alone because we
have two existing 3GP media assets that were falsely detected as MP4.
https://danbooru.donmai.us/forum_topics/22356https://github.com/mozilla/gecko-dev/blob/master/toolkit/components/mediasniffer/nsMediaSniffer.cpp#L78https://mimesniff.spec.whatwg.org/#signature-for-mp4
Don't allow uploading videos with unsupported video codecs.
The only video codecs we allow for MP4 files are H.264 and VP9. Other
codecs, including H.265 (aka HEVC), MPEG-4 part 2, and AV1, are
disallowed because they're not universally supported by browsers.
Firefox doesn't support H.265 or MPEG-4 part 2, and Safari doesn't
support AV1.
Additionally, don't allow videos with multiple video tracks, multiple
audio tracks, or no video tracks. Multiple video and audio tracks are
disallowed because they're rare and for moderation purposes, we don't
want people hiding content in extra tracks.
These restrictions really only apply to MP4 videos, since WebM files
don't support multiple video or audio tracks and only support a limited
number of codecs (VP8 and VP9 for videos, Vorbis and Opus for audio).
There are currently 22 posts with unsupported video codecs:
* https://danbooru.donmai.us/posts?tags=video+is:mp4+-exif:Track1:CompressorID=avc1+-exif:Track2:CompressorID=avc1+-exif:Track1:CompressorID=vp09+-exif:Track2:CompressorID=vp09 # AVC1 is H.264
There is one post that has multiple audio tracks:
* https://danbooru.donmai.us/posts/2382057
