* Make it so replacing a post doesn't generate a dummy upload as a side effect.
* Make it so you can't replace a post with itself (the post should be regenerated instead).
* Refactor uploads and replacements to save the ugoira frame data when
the MediaAsset is created, not when the post is created. This way it's
possible to view the ugoira before the post is created.
* Make `download_file!` in the Pixiv source strategy return a MediaFile
with the ugoira frame data already attached to it, instead of returning it
in the `data` field then passing it around separately in the `context`
field of the upload.
Move more of the file-handling logic from UploadService and
StorageManager into MediaAsset. This is part of refactoring posts and
uploads to allow multiple images per post.
Automatically tag animated_gif and animated_png when a post is edited.
Add them back if the user tries to remove them from an animated post,
or remove them if the user tries to add them to a non-animated post.
Before we added these tags at upload time, but it was possible for users
to remove them after upload, or to incorrectly add them to non-animated
posts. They were added at upload time because we couldn't afford to open
the file and parse the metadata on every tag edit. Now that we save the
metadata in the database, we can do this.
This also makes it so you can't tag ugoira on non-ugoira files.
Known bug: it's possible to have an animated GIF where every frame is
identical. Post #3770975 is an example. This will be detected as an
animated GIF even though visually it doesn't appear to be animated.
Fixes#4041: Animated_gif tag not added to preprocessed uploads
Add a model for storing image and video metadata for uploaded files.
Metadata is extracted using ExifTool. You will need to install ExifTool
after this commit. ExifTool 12.22 is the minimum required version
because we use the `--binary` option, which was added in this release.
The MediaMetadata model is separate from the MediaAsset model because
some files contain tons of metadata, and most of it is non-essential.
The MediaAsset model represents an uploaded file and contains essential
metadata, like the file's size and type, while the MediaMetadata model
represents all the other non-essential metadata associated with a file.
Metadata is stored as a JSON column in the database.
ExifTool returns all the file's metadata, not just the EXIF metadata.
EXIF is one of several types of image metadata, hence why we call
it MediaMetadata instead of EXIFMetadata.
Remove the Downloads::File class. Move download methods to
Danbooru::Http instead. This means that:
* HTTParty has been replaced with http.rb for downloading files.
* Downloading is no longer tightly coupled to source strategies. Before
Downloads::File tried to automatically look up the source and download
the full size image instead if we gave it a sample url. Now we can
do plain downloads without source strategies altering the url.
* The Cloudflare Polish check has been changed from checking for a
Cloudflare IP to checking for the CF-Polished header. Looking up the
list of Cloudflare IPs was slow and flaky during testing.
* The SSRF protection code has been factored out so it can be used for
normal http requests, not just for downloads.
* The Webmock gem can be removed, since it was only used for stubbing
out certain HTTParty requests in the download tests. The Webmock gem
is buggy and caused certain tests to fail during CI.
* The retriable gem can be removed, since we no longer autoretry failed
downloads. We assume that if a download fails once then retrying
probably won't help.
ref: https://danbooru.donmai.us/forum_topics/16935.
Bug: sample images were being generated to be at most 850px width *and*
850px tall. They're supposed to be at most 850px wide with unlimited height.
* Move image thumbnail generation code to MediaFile::Image.
* Move video thumbnail generation code to MediaFile::Video.
* Move ugoira->webm conversion code to MediaFile::Ugoira.
This separates thumbnail generation from the upload process so that it's
possible to generate thumbnails outside of uploads.
* Add MediaFile abstraction. A MediaFile represents an image or video file.
* Move filetype detection and dimension parsing code from uploads to MediaFile.
* Fix corrupted image detection. We were shelling out to vips and trying
to grep for error messages, but the error message for jpeg files changed.
Now we load the file in ruby vips, which raises an error on failure.
* Don't attempt to redownload corrupted images. If a download completes
without any errors yet the downloaded file is corrupt, then something is
wrong at the source and redownloading is unlikely to help. Let the
upload fail and the user retry if necessary.
* Validate that all uploads are uncorrupted, including files uploaded
from a computer, not just files uploaded from a source.
Bug: Uploading bad pixiv id images failed because the pixiv strategy
raised a BadIDError exception when the upload service checked for the
ugoira frame data.
* Rename Upload#download_for_upload to #get_file_for_upload.
* Fix #get_file_for_upload to raise error if no file or source url was given.
* Fix javascript upload validation to disallow submitting form if file is
not present and the source is not an url.
