Bug: the uploads page showed a remote size of 146 bytes for Pixiv uploads.
Cause: we didn't spoof the Referer header when making the HEAD request
for the image, causing Pixiv to return a 403 error.
Also fix the case where the Content-Length header is absent.
Remove the Downloads::File class. Move download methods to
Danbooru::Http instead. This means that:
* HTTParty has been replaced with http.rb for downloading files.
* Downloading is no longer tightly coupled to source strategies. Before
Downloads::File tried to automatically look up the source and download
the full size image instead if we gave it a sample url. Now we can
do plain downloads without source strategies altering the url.
* The Cloudflare Polish check has been changed from checking for a
Cloudflare IP to checking for the CF-Polished header. Looking up the
list of Cloudflare IPs was slow and flaky during testing.
* The SSRF protection code has been factored out so it can be used for
normal http requests, not just for downloads.
* The Webmock gem can be removed, since it was only used for stubbing
out certain HTTParty requests in the download tests. The Webmock gem
is buggy and caused certain tests to fail during CI.
* The retriable gem can be removed, since we no longer autoretry failed
downloads. We assume that if a download fails once then retrying
probably won't help.
ref: https://danbooru.donmai.us/forum_topics/16935.
Bug: sample images were being generated to be at most 850px width *and*
850px tall. They're supposed to be at most 850px wide with unlimited height.
* Move image thumbnail generation code to MediaFile::Image.
* Move video thumbnail generation code to MediaFile::Video.
* Move ugoira->webm conversion code to MediaFile::Ugoira.
This separates thumbnail generation from the upload process so that it's
possible to generate thumbnails outside of uploads.
When doing a tag search, we have to be careful about which user we're
running the search as because the results depend on the current user.
Specifically, things like private favorites, private favorite groups,
post votes, saved searches, and flagger names depend on the user's
permissions, and whether non-safe or deleted posts are filtered out
depend on whether the user has safe mode on or the hide deleted posts
setting enabled.
* Refactor internal searches to explicitly state whether they're
running as the system user (DanbooruBot) or as the current user.
* Explicitly pass in the current user to PostQueryBuilder instead of
implicitly relying on the CurrentUser global.
* Get rid of CurrentUser.admin_mode? (used to ignore the hide deleted
post setting) and CurrentUser.without_safe_mode (used to ignore safe
mode).
* Change the /counts/posts.json endpoint to ignore safe mode and the
hide deleted posts settings when counting posts.
* Fix searches not correctly overriding the hide deleted posts setting
when multiple status: metatags were used (e.g. `status:banned status:active`)
* Fix fast_count not respecting the hide deleted posts setting when the
status:banned metatag was used.
* Add MediaFile abstraction. A MediaFile represents an image or video file.
* Move filetype detection and dimension parsing code from uploads to MediaFile.
* Fix corrupted image detection. We were shelling out to vips and trying
to grep for error messages, but the error message for jpeg files changed.
Now we load the file in ruby vips, which raises an error on failure.
* Don't attempt to redownload corrupted images. If a download completes
without any errors yet the downloaded file is corrupt, then something is
wrong at the source and redownloading is unlikely to help. Let the
upload fail and the user retry if necessary.
* Validate that all uploads are uncorrupted, including files uploaded
from a computer, not just files uploaded from a source.
The belongs_to_creator macro was used to initialize the creator_id field
to the CurrentUser. This made tests complicated because it meant you had
to create and set the current user every time you wanted to create an
object, when lead to the current user being set over and over again. It
also meant you had to constantly be aware of what the CurrentUser was in
many different contexts, which was often confusing. Setting creators
explicitly simplifies everything greatly.
* Switch CloudflareService from HttpartyCache to Danbooru::Http.
* Purge cached urls from Cloudflare when a post is replaced and the md5
doesn't change. This happens when a corrupted image is replaced or
thumbnails are regenerated. Before we purged urls when a post was
expunged, which was unneeded because those urls can expire naturally.
It was also wrong because the subdomains were hardcoded, the urls used
http:// instead of https://, and we didn't account for tagged urls.
Bug: Uploading bad pixiv id images failed because the pixiv strategy
raised a BadIDError exception when the upload service checked for the
ugoira frame data.
Bug: Notes weren't rescaled when the 'final_source' field was given
during replacement.
The cause was that the notes were rescaled after the source was saved,
but saving the source clobbered `image_{width,height}_before_last_save`
inside `rescale_notes`.
Regressed in b0c2ddba.
* Rename Upload#download_for_upload to #get_file_for_upload.
* Fix #get_file_for_upload to raise error if no file or source url was given.
* Fix javascript upload validation to disallow submitting form if file is
not present and the source is not an url.
* On the /uploads/new page, instead of just showing a "This post has
probably already been uploaded" message, show the actual thumbnails of
posts having the same source as what the user is trying to upload.
* Move the iqdb results section up top, beside the related posts section.
Also makes it so that using the bookmarklet always triggers async upload
preprocessing. Before it was only triggered when the source passed a
dupe check, but that check was inaccurate (#3873).
