Fix temp files generated during the upload process not being cleaned up quickly enough. This included
downloaded files, generated preview images, and Ugoira video conversions.
Before we relied on `Tempfile` cleaning up files automatically. But this only happened when the
Tempfile object was garbage collected, which could take a long time. In the meantime we could have
hundreds of megabytes of temp files hanging around.
The fix is to explicitly close temp files when we're done with them. But the standard `Tempfile`
class doesn't immediately delete the file when it's closed. So we also have to introduce a
Danbooru::Tempfile wrapper that deletes the tempfile as soon as it's closed.
At first we rounded loudness values to 4 decimal places to make them
easier to compare. This meant the lowest level was 0.0001, or -80 dB,
but it's possible for volume levels to go even lower than that.
For videos with sound, save information about audio volume levels in the
media asset's metadata. These values are stored:
* FFmpeg:AudioPeakLoudness The peak loudness of the audio track, from 0.0 (silent) to 1.0 (max volume)
* FFmpeg:AudioAverageLoudness The average loudness of the audio track, from 0.0 (silent) to 1.0 (max volume).
* FFmpeg:AudioLoudnessRange The difference between the quietest and loudest sounds in the audio track (in decibels).
* FFmpeg:AudioSilencePercentage The percentage of the video that is silent (1.0 is completely silent, 0.5 is 50% silence, 0.0 is no silence).
These values are calculated based on the EBU R 128 standard, using the ffmpeg command below:
ffmpeg -i file.mp4 -af silencedetect=duration=0.05:noise=0.0001,ebur128=metadata=1:peak=true:dualmono=true -f null /dev/null
See the links below for details:
* https://en.wikipedia.org/wiki/EBU_R_128
* https://www.ffmpeg.org/ffmpeg-filters.html#ebur128-1
* https://tech.ebu.ch/loudness
* https://tech.ebu.ch/docs/tech/tech3341.pdf
Fix .webm files not including the `FFmpeg:VideoBitRate` and `FFmpeg:AudioBitRate`
fields in the media_metadata table. This was because the .webm format
doesn't include the video or audio bit rates in the metadata, and
ffprobe doesn't calculate them either, so we have to calculate them
ourselves by hand.
Fixup for 523d7afdd.
Include the following metadata tags for videos:
* FFmpeg:MajorBrand (e.g. mp42)
* FFmpeg:PixFmt (e.g. yuv420p. Indicates bit depth and color subsampling mode)
* FFmpeg:FrameCount (e.g. total number of frames in the video)
* FFmpeg:VideoCodec (e.g. h264)
* FFmpeg:VideoProfile (e.g. Main)
* FFmpeg:AudioCodec (e.g. AAC)
* FFmpeg:AudioProfile (e.g. LC)
* FFmpeg:AudioLayout (e.g. stereo)
* FFmpeg:AudioBitRate (e.g. 128kb/s)
If a media asset is corrupt, include the error message from libvips or
ffmpeg in the "Vips:Error" or "FFmpeg:Error" fields in the media
metadata table.
Corrupt files can't be uploaded nowadays, but they could be in the past,
so we have some old corrupted files that we can't generate thumbnails
for. This lets us mark these files in the metadata so they're findable
with the tag search `exif:Vips:Error`.
Known bug: Vips has a single global error buffer that is shared between
threads and that isn't cleared between operations. So we can't reliably
get the actual error message because it may pick up errors from other
threads, or from previous operations in the same thread.
Don't allow uploading videos with unsupported video codecs.
The only video codecs we allow for MP4 files are H.264 and VP9. Other
codecs, including H.265 (aka HEVC), MPEG-4 part 2, and AV1, are
disallowed because they're not universally supported by browsers.
Firefox doesn't support H.265 or MPEG-4 part 2, and Safari doesn't
support AV1.
Additionally, don't allow videos with multiple video tracks, multiple
audio tracks, or no video tracks. Multiple video and audio tracks are
disallowed because they're rare and for moderation purposes, we don't
want people hiding content in extra tracks.
These restrictions really only apply to MP4 videos, since WebM files
don't support multiple video or audio tracks and only support a limited
number of codecs (VP8 and VP9 for videos, Vorbis and Opus for audio).
There are currently 22 posts with unsupported video codecs:
* https://danbooru.donmai.us/posts?tags=video+is:mp4+-exif:Track1:CompressorID=avc1+-exif:Track2:CompressorID=avc1+-exif:Track1:CompressorID=vp09+-exif:Track2:CompressorID=vp09 # AVC1 is H.264
There is one post that has multiple audio tracks:
* https://danbooru.donmai.us/posts/2382057
Features of AVIF include:
* Lossless and lossy compression.
* High dynamic range (HDR) images
* Wide color gamut images (i.e. 10- and 12-bit color depths)
* Transparency (through alpha planes).
* Animations (with an optional cover image).
* Auxiliary image sequences, where the file contains a single primary
image and a short secondary video, like Apple's Live Photos.
* Metadata rotation, mirroring, and cropping.
The AVIF format is still relatively new and some of these features aren't well
supported by browsers or other software:
* Animated AVIFs aren't supported by Firefox or by libvips.
* HDR images aren't supported by Firefox.
* Rotated, mirrored, and cropped AVIFs aren't supported by Firefox or Chrome.
* Image grids, where the file contains multiple images that are tiled
together into one big image, aren't supported by Firefox.
* AVIF as a whole has only been supported for a year or two by Chrome
and Firefox, and less than a year by Safari.
For these reasons, only basic AVIFs that don't use animation, rotation,
cropping, or image grids can be uploaded.
Fix how the duration of videos and animated GIFs / PNGs is calculated.
If we can't determine the duration from the file metadata, then play the
entire video or animation back using FFmpeg and scrape the duration and
frame count.
This is necessary for things like WebM files where the duration metadata
is optional, or animated GIFs and PNGs that don't have a duration field
in the metadata, only a frame count and a sequence of frame delays.
Fix certain animated PNGs returning NaN as the duration because the
frame rate was being reported as "0/0" by FFMpeg. This happens when the
animation has zero delay between frames. This is supposed to mean a PNG
with an infinitely fast frame rate, but in practice browsers limit it to
around 10FPS. The exact frame rate browsers will use is unknown and
implementation defined.
Add methods to MediaFile to calculate the duration, frame count, and
frame rate of animated GIFs, PNGs, Ugoiras, and videos.
Some considerations:
* It's possible to have a GIF or PNG that's technically animated but
just has one frame. These are treated as non-animated images.
* It's possible to have an animated GIF that has an unspecified
frame rate. In this case we assume the frame rate is 10 FPS; this is
browser dependent and may not be correct.
* Animated GIFs, PNGs, and Ugoiras all support variable frame rates.
Technically, each frame has a separate delay, and the delays can be
different frame-to-frame. We report only the average frame rate.
* Getting the duration of an APNG is surprisingly hard. Most tools don't
have good support for APNGs since it's a rare and non-standardized
format. The best we can do is get the frame count using ExifTool and the
frame rate using ffprobe, then calculate the duration from that.
Generate smart previews as .png so we don't suffer recompression losses
when we convert the preview frame from a full size image down to a
150x150 .jpg thumbnail.
