Remove the last remaining uses of the PixivUgoiraFrameData model. As of
32bfb8407, Ugoira frame data is now stored in the MediaMetadata model,
under the `Ugoira:FrameDelays` EXIF field.
The pixiv_ugoira_frame_data table still exists, but it can be removed
after this commit is deployed.
Fixes#5264: Error when replacing with ugoira.
Remove the `image_url` method from source strategies. This method would
return only the first image if a source had multiple images. The
`image_urls` method should be used instead. Tests were the main place
that still used `image_url` instead of `image_urls`.
Also make post replacements return an error if replacing with a source
that contains multiple images, instead of just blindly replacing the
post with the first image in the source.
* Fix error when uploading non-ugoira files.
* Fix sample image URLs not being rewritten to full images correctly. We
have to get the full image URL from the API because given an
/img-master/ URL, we don't know what the original file extension is.
* Make it so replacing a post doesn't generate a dummy upload as a side effect.
* Make it so you can't replace a post with itself (the post should be regenerated instead).
* Refactor uploads and replacements to save the ugoira frame data when
the MediaAsset is created, not when the post is created. This way it's
possible to view the ugoira before the post is created.
* Make `download_file!` in the Pixiv source strategy return a MediaFile
with the ugoira frame data already attached to it, instead of returning it
in the `data` field then passing it around separately in the `context`
field of the upload.
* Skip Nijie tests because they fail a lot due to Nijie rate limiting us.
* Skip ArtStation downloads tests because they sometimes return different file sizes.
* Fix random duplicate favgroup errors because favgroup names weren't random enough.
Fix the test suite failing when trying to run it in the default state
with no config file or API keys configured. Most source sites require
API keys or login credentials to be set in order to work. Skip these
tests when credentials aren't configured.
Fix the Pixiv API no longer working by rewriting the Pixiv strategy to
use the Ajax API instead of the mobile API.
Before we could authenticate in the mobile API by using the OAuth 2.0
grant_type=password authentication flow. This no longer works. Now it
requires logging in through a HTML page, which is protected by Google
reCaptcha. This makes using the mobile API infeasible.
Instead we switch to the Ajax API, which only needs a PHPSESSID to
authenticate. This can be obtained by logging in manually and using the
devtools to extract the cookie.
This also temporarily removes support for Pixiv novels. This should be
moved to a separate source strategy.
ArtStation is apparently returning images with smaller filesizes for
/4k/ images now. It doesn't look like we can get the old images, so just
update the tests.
Remove the Downloads::File class. Move download methods to
Danbooru::Http instead. This means that:
* HTTParty has been replaced with http.rb for downloading files.
* Downloading is no longer tightly coupled to source strategies. Before
Downloads::File tried to automatically look up the source and download
the full size image instead if we gave it a sample url. Now we can
do plain downloads without source strategies altering the url.
* The Cloudflare Polish check has been changed from checking for a
Cloudflare IP to checking for the CF-Polished header. Looking up the
list of Cloudflare IPs was slow and flaky during testing.
* The SSRF protection code has been factored out so it can be used for
normal http requests, not just for downloads.
* The Webmock gem can be removed, since it was only used for stubbing
out certain HTTParty requests in the download tests. The Webmock gem
is buggy and caused certain tests to fail during CI.
* The retriable gem can be removed, since we no longer autoretry failed
downloads. We assume that if a download fails once then retrying
probably won't help.
This was a hack to deal with the Cloudflare check sometimes being slow
or timing out during tests. The call to https://api.cloudflare.com/client/v4/ips
could hang if there were IPv6 connectivity problems. If this happens, make
sure that IPv6 is configured properly and that `curl -v --http1.1 -6 https://api.cloudflare.com/client/v4/ips`
works.
Fixes the banned IP check not being applied when sending the HEAD
request for is_cloudflare?.
Also fixes the `#size` method not using the uncached url (which meant
the bookmarklet could report the wrong filesize on artstation uploads).
Fail loudly if we forget to whitelist a param instead of silently
ignoring it.
misc models: convert to strong params.
artist commentaries: convert to strong params.
* Disallow changing or setting post_id to a nonexistent post.
artists: convert to strong params.
* Disallow setting `is_banned` in create/update actions. Changing it
this way instead of with the ban/unban actions would leave the artist in
a partially banned state.
bans: convert to strong params.
* Disallow changing the user_id after the ban has been created.
comments: convert to strong params.
favorite groups: convert to strong params.
news updates: convert to strong params.
post appeals: convert to strong params.
post flags: convert to strong params.
* Disallow users from setting the `is_deleted` / `is_resolved` flags.
ip bans: convert to strong params.
user feedbacks: convert to strong params.
* Disallow users from setting `disable_dmail_notification` when creating feedbacks.
* Disallow changing the user_id after the feedback has been created.
notes: convert to strong params.
wiki pages: convert to strong params.
* Also fix non-Builders being able to delete wiki pages.
saved searches: convert to strong params.
pools: convert to strong params.
* Disallow setting `post_count` or `is_deleted` in create/update actions.
janitor trials: convert to strong params.
post disapprovals: convert to strong params.
* Factor out quick-mod bar to shared partial.
* Fix quick-mod bar to use `Post#is_approvable?` to determine visibility
of Approve button.
dmail filters: convert to strong params.
password resets: convert to strong params.
user name change requests: convert to strong params.
posts: convert to strong params.
users: convert to strong params.
* Disallow setting password_hash, last_logged_in_at, last_forum_read_at,
has_mail, and dmail_filter_attributes[user_id].
* Remove initialize_default_image_size (dead code).
uploads: convert to strong params.
* Remove `initialize_status` because status already defaults to pending
in the database.
tag aliases/implications: convert to strong params.
tags: convert to strong params.
forum posts: convert to strong params.
* Disallow changing the topic_id after creating the post.
* Disallow setting is_deleted (destroy/undelete actions should be used instead).
* Remove is_sticky / is_locked (nonexistent attributes).
forum topics: convert to strong params.
* merges https://github.com/evazion/danbooru/tree/wip-rails-5.1
* lock pg gem to 0.21 (1.0.0 is incompatible with rails 5.1.4)
* switch to factorybot and change all references
Co-authored-by: r888888888 <r888888888@gmail.com>
Co-authored-by: evazion <noizave@gmail.com>
add diffs
* Change `http_get_streaming` to write the output file directly,
instead of taking a callback.
* Track the filesize as the download progresses and abort when it
exceeds the limit.
* Don't save the Content-Type (it's not used anywhere).
