If a media asset is corrupt, include the error message from libvips or
ffmpeg in the "Vips:Error" or "FFmpeg:Error" fields in the media
metadata table.
Corrupt files can't be uploaded nowadays, but they could be in the past,
so we have some old corrupted files that we can't generate thumbnails
for. This lets us mark these files in the metadata so they're findable
with the tag search `exif:Vips:Error`.
Known bug: Vips has a single global error buffer that is shared between
threads and that isn't cleared between operations. So we can't reliably
get the actual error message because it may pick up errors from other
threads, or from previous operations in the same thread.
Fix it so that trying to regenerate AI tags for a Flash file doesn't
fail because Flash files have no image preview.
Also let `MediaFile.open` take a block argument.
Fix a bug where MP4 files with major brand "iso4" weren't detected as
MP4, so they couldn't be uploaded.
This switches our MP4 detection code to something very similar to Firefox's
MP4 sniffing algorithm. Ours is slightly wrong because a) we only check
the major_brand, not the minor_brands, and b) we falsely detect certain 3GP
videos as MP4. 3GP is a very similar format to MP4, close enough that it
can be played by Chrome (but not Firefox), but it's technically not MP4
and should not have a .mp4 file extension. We leave it alone because we
have two existing 3GP media assets that were falsely detected as MP4.
https://danbooru.donmai.us/forum_topics/22356https://github.com/mozilla/gecko-dev/blob/master/toolkit/components/mediasniffer/nsMediaSniffer.cpp#L78https://mimesniff.spec.whatwg.org/#signature-for-mp4
Don't allow uploading videos with unsupported video codecs.
The only video codecs we allow for MP4 files are H.264 and VP9. Other
codecs, including H.265 (aka HEVC), MPEG-4 part 2, and AV1, are
disallowed because they're not universally supported by browsers.
Firefox doesn't support H.265 or MPEG-4 part 2, and Safari doesn't
support AV1.
Additionally, don't allow videos with multiple video tracks, multiple
audio tracks, or no video tracks. Multiple video and audio tracks are
disallowed because they're rare and for moderation purposes, we don't
want people hiding content in extra tracks.
These restrictions really only apply to MP4 videos, since WebM files
don't support multiple video or audio tracks and only support a limited
number of codecs (VP8 and VP9 for videos, Vorbis and Opus for audio).
There are currently 22 posts with unsupported video codecs:
* https://danbooru.donmai.us/posts?tags=video+is:mp4+-exif:Track1:CompressorID=avc1+-exif:Track2:CompressorID=avc1+-exif:Track1:CompressorID=vp09+-exif:Track2:CompressorID=vp09 # AVC1 is H.264
There is one post that has multiple audio tracks:
* https://danbooru.donmai.us/posts/2382057
Fix StatementInvalid exception when uploading https://files.catbox.moe/vxoe2p.mp4.
This was a result of multiple bugs:
* First, generating thumbnails for the video failed. This was because
the video uses the AV1 codec, which FFmpeg failed to decode. It failed
because our version of FFmpeg was built without the `--enable-libdav1d`
flag, so it uses the builtin AV1 decoder, which apparently can't
handle this particular video (it spews a bunch of errors about "Failed
to get pixel format" and "missing sequence header" and "failed to get
reference frame").
* Because generating the thumbnails failed, an exception was raised. We
tried to save the error message in the upload_media_assets.error
field. However, this also failed because the error message was 77kb
long (it contained the entire output of the ffmpeg command), but the
`upload_media_assets` table had a btree index on the `error` column,
which meant the maximum length of the error column was limited to
~2.7kb. This lead to a StatementInvalid exception being raised.
* Because the StatementInvalid exception was raised while we were trying
to set the upload media asset's status to `failed`, the upload was
left stuck in the `processing` state rather than being set to the
`failed` state.
* Because the upload was stuck in the `processing` state, the upload
page would hang forever waiting for the upload to complete.
The fixes are to:
* Build FFmpeg with `--enable-libdav1d` to use libdav1d for decoding AV1
videos instead of the builtin AV1 decoder.
* Remove the index on the `upload_media_assets.error` column so that
setting overly long error messages won't fail.
* Catch unexpected exceptions in ProcessUploadMediaAssetJob so we can
mark uploads as failed, even if `process_upload!` itself fails because
it raises an unexpected exception inside its own exception handler.
* Check that the video is playable with `MediaFile::Video#is_corrupt?` before
allowing it to be uploaded. This way we can return a better error
message if we can't generate thumbnails because the video isn't
playable. This requires decoding the entire video, so it means uploads
may take several seconds longer for long videos. It's also a security
risk in case ffmpeg has any bugs.
* Define `MediaAsset#preview!` as raising an exception on error, so
it's clear that generating thumbnails can fail. Define `MediaAsset#preview`
as returning nil on error for when we don't care about the cause of
the error.
Add ability to upload .webp images.
Animated WebP images aren't supported. This is because they aren't
supported by FFmpeg yet[1], so generating thumbnails and samples for
them would be more complicated than for other formats.
[1]: https://trac.ffmpeg.org/ticket/4907
Features of AVIF include:
* Lossless and lossy compression.
* High dynamic range (HDR) images
* Wide color gamut images (i.e. 10- and 12-bit color depths)
* Transparency (through alpha planes).
* Animations (with an optional cover image).
* Auxiliary image sequences, where the file contains a single primary
image and a short secondary video, like Apple's Live Photos.
* Metadata rotation, mirroring, and cropping.
The AVIF format is still relatively new and some of these features aren't well
supported by browsers or other software:
* Animated AVIFs aren't supported by Firefox or by libvips.
* HDR images aren't supported by Firefox.
* Rotated, mirrored, and cropped AVIFs aren't supported by Firefox or Chrome.
* Image grids, where the file contains multiple images that are tiled
together into one big image, aren't supported by Firefox.
* AVIF as a whole has only been supported for a year or two by Chrome
and Firefox, and less than a year by Safari.
For these reasons, only basic AVIFs that don't use animation, rotation,
cropping, or image grids can be uploaded.
Calculate the dimensions of thumbnails ourselves instead of letting
libvips calculate them for us. This way we know the exact size of
thumbnails, so we can set the right width and height for <img> tags. If
we let libvips calculate thumbnail sizes for us, then we can't predict
the exact size of thumbnails, because sometimes libvips rounds numbers
differently than us.
* Make it so replacing a post doesn't generate a dummy upload as a side effect.
* Make it so you can't replace a post with itself (the post should be regenerated instead).
* Refactor uploads and replacements to save the ugoira frame data when
the MediaAsset is created, not when the post is created. This way it's
possible to view the ugoira before the post is created.
* Make `download_file!` in the Pixiv source strategy return a MediaFile
with the ugoira frame data already attached to it, instead of returning it
in the `data` field then passing it around separately in the `context`
field of the upload.
Add methods to MediaFile to calculate the duration, frame count, and
frame rate of animated GIFs, PNGs, Ugoiras, and videos.
Some considerations:
* It's possible to have a GIF or PNG that's technically animated but
just has one frame. These are treated as non-animated images.
* It's possible to have an animated GIF that has an unspecified
frame rate. In this case we assume the frame rate is 10 FPS; this is
browser dependent and may not be correct.
* Animated GIFs, PNGs, and Ugoiras all support variable frame rates.
Technically, each frame has a separate delay, and the delays can be
different frame-to-frame. We report only the average frame rate.
* Getting the duration of an APNG is surprisingly hard. Most tools don't
have good support for APNGs since it's a rare and non-standardized
format. The best we can do is get the frame count using ExifTool and the
frame rate using ffprobe, then calculate the duration from that.
Add a model for storing image and video metadata for uploaded files.
Metadata is extracted using ExifTool. You will need to install ExifTool
after this commit. ExifTool 12.22 is the minimum required version
because we use the `--binary` option, which was added in this release.
The MediaMetadata model is separate from the MediaAsset model because
some files contain tons of metadata, and most of it is non-essential.
The MediaAsset model represents an uploaded file and contains essential
metadata, like the file's size and type, while the MediaMetadata model
represents all the other non-essential metadata associated with a file.
Metadata is stored as a JSON column in the database.
ExifTool returns all the file's metadata, not just the EXIF metadata.
EXIF is one of several types of image metadata, hence why we call
it MediaMetadata instead of EXIFMetadata.
* Move image thumbnail generation code to MediaFile::Image.
* Move video thumbnail generation code to MediaFile::Video.
* Move ugoira->webm conversion code to MediaFile::Ugoira.
This separates thumbnail generation from the upload process so that it's
possible to generate thumbnails outside of uploads.
* Add MediaFile abstraction. A MediaFile represents an image or video file.
* Move filetype detection and dimension parsing code from uploads to MediaFile.
