Commit Graph

13 Commits

Author SHA1 Message Date
evazion
26ad844bbe downloads: refactor Downloads::File into Danbooru::Http.
Remove the Downloads::File class. Move download methods to
Danbooru::Http instead. This means that:

* HTTParty has been replaced with http.rb for downloading files.

* Downloading is no longer tightly coupled to source strategies. Before
  Downloads::File tried to automatically look up the source and download
  the full size image instead if we gave it a sample url. Now we can
  do plain downloads without source strategies altering the url.

* The Cloudflare Polish check has been changed from checking for a
  Cloudflare IP to checking for the CF-Polished header. Looking up the
  list of Cloudflare IPs was slow and flaky during testing.

* The SSRF protection code has been factored out so it can be used for
  normal http requests, not just for downloads.

* The Webmock gem can be removed, since it was only used for stubbing
  out certain HTTParty requests in the download tests. The Webmock gem
  is buggy and caused certain tests to fail during CI.

* The retriable gem can be removed, since we no longer autoretry failed
  downloads. We assume that if a download fails once then retrying
  probably won't help.
2020-06-20 00:20:39 -05:00
evazion
ab1931a920 tests: fix artstation download tests.
Fix ArtStation download tests that were broken when 266e4054 added the
ability to download /4k/ files.
2020-06-10 19:45:56 -05:00
evazion
266e4054b0 Fix #4293: ArtStation: use 4k images.
Also fixes #4290 (Image replacements: undefined method hostname for nil:NilClass)
2020-03-03 23:01:29 -06:00
evazion
f4e08ef30d Downloads::File: fix SSRF inside is_cloudflare? (#2498).
Fixes the banned IP check not being applied when sending the HEAD
request for is_cloudflare?.

Also fixes the `#size` method not using the uncached url (which meant
the bookmarklet could report the wrong filesize on artstation uploads).
2018-09-19 20:11:53 -05:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
evazion
d089be9f8a tests: fix upload tests. 2018-03-20 19:49:58 -05:00
evazion
c9eee7e4d4 Fix #3528: Prevent CloudFlare from altering images. 2018-02-24 13:42:00 -06:00
Albert Yi
d8340f83db Revert "Fix #3528: Add method to prevent image-hosting CloudFlare sites from altering the image" 2018-02-21 17:20:13 -08:00
evazion
263fd0eef1 Fix #3528: Prevent CloudFlare from altering images. 2018-02-17 11:43:27 -06:00
r888888888
3c524aa5e6 fix bug with source::site 2017-11-17 17:26:33 -08:00
evazion
b3e533f3ff Add download rewrite for https://$artist.artstation.com/projects/$id. 2017-06-05 14:03:51 -05:00
r888888888
7a3d1c7d1d dont download original artstation images if they dont exist 2017-02-07 13:40:19 -08:00
r888888888
5c16e9ce48 add download strategy for artstation 2017-02-06 12:03:00 -08:00