danbooru::http: factor out Cloudflare Polish bypassing.

* Factor out the Cloudflare Polish bypass code to a standalone feature.

* Add `http_downloader` method to the base source strategy. This is a
  HTTP client that should be used for downloading images or making
  requests to images. This client ensures that referrer spoofing and
  Cloudflare bypassing are performed.

This fixes a bug with the upload page reporting the polished filesize
instead of the original filesize when uploading ArtStation images.
This commit is contained in:
evazion
2020-06-24 21:33:10 -05:00
parent d3bb5c67ee
commit 5af50b7fcd
5 changed files with 47 additions and 21 deletions

View File

@@ -155,7 +155,7 @@ module Sources
# the api_response wasn't available because it's a deleted post.
elsif post_md5.present?
%w[jpg png gif].find { |ext| http_exists?("https://#{site_name}/image/#{post_md5}.#{ext}", headers) }
%w[jpg png gif].find { |ext| http_exists?("https://#{site_name}/image/#{post_md5}.#{ext}") }
else
nil