Commit Graph

41 Commits

Author SHA1 Message Date
evazion
5604ab0079 pixiv: remove fanbox support.
This is broken and it needs to be rewritten as a separate source
strategy anyway.
2020-06-21 11:59:51 -05:00
evazion
26ad844bbe downloads: refactor Downloads::File into Danbooru::Http.
Remove the Downloads::File class. Move download methods to
Danbooru::Http instead. This means that:

* HTTParty has been replaced with http.rb for downloading files.

* Downloading is no longer tightly coupled to source strategies. Before
  Downloads::File tried to automatically look up the source and download
  the full size image instead if we gave it a sample url. Now we can
  do plain downloads without source strategies altering the url.

* The Cloudflare Polish check has been changed from checking for a
  Cloudflare IP to checking for the CF-Polished header. Looking up the
  list of Cloudflare IPs was slow and flaky during testing.

* The SSRF protection code has been factored out so it can be used for
  normal http requests, not just for downloads.

* The Webmock gem can be removed, since it was only used for stubbing
  out certain HTTParty requests in the download tests. The Webmock gem
  is buggy and caused certain tests to fail during CI.

* The retriable gem can be removed, since we no longer autoretry failed
  downloads. We assume that if a download fails once then retrying
  probably won't help.
2020-06-20 00:20:39 -05:00
evazion
2d05004bef tests: don't cache pixiv sessions. 2020-06-11 00:47:12 -05:00
evazion
d6b266514b tests: disable known broken pixiv fanbox tests. 2020-06-10 18:21:44 -05:00
evazion
309821bf73 rubocop: fix various style issues. 2019-12-22 21:23:37 -06:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
Albert Yi
77854349e5 testing 2018-07-26 18:11:19 -07:00
Albert Yi
5ae37597cd fixes #3728 2018-05-25 13:24:49 -07:00
Albert Yi
6839249e9d add logic for persisting pixiv sessions in tests 2018-05-10 13:59:57 -07:00
Albert Yi
5bca31bad1 add retry monkey patch for mechanize 2018-05-09 16:45:03 -07:00
Albert Yi
e23814be92 fix pixiv agent shutdown 2018-05-09 16:13:47 -07:00
Albert Yi
d4a2521eec reset connection for pixiv tests 2018-05-09 14:59:18 -07:00
evazion
181a906766 Fix #3695: Bookmarklet breaks on pixiv fanbox direct image links. 2018-05-05 12:21:11 -05:00
evazion
d089be9f8a tests: fix upload tests. 2018-03-20 19:49:58 -05:00
evazion
a923a64220 tests: fix pixiv profile image test. 2018-01-20 14:51:05 -06:00
evazion
53f7207716 tests: remove bad_id pixiv test. 2018-01-20 14:50:44 -06:00
evazion
0b5ed163fe Fix #3395: fix pixiv API exception when uploading bad pixiv id images.
Fixes an exception when attempting to upload a Pixiv image from a
deleted work. The download strategy tries to fetch the source data in
the course of rewriting the URL, which fails if the work has been
deleted from Pixiv.

Raise a BadIDError and leave the URL as-is (don't rewrite it).
2017-11-21 14:40:39 -06:00
evazion
5ba63733d3 download/pixiv_test.rb: fix broken pixiv download tests.
This post was deleted:

  http://www.pixiv.net/member_illust.php?mode=medium&illust_id=46304614

Switch to this instead:

  http://www.pixiv.net/member_illust.php?mode=medium&illust_id=46324488
2017-11-21 13:48:07 -06:00
evazion
a14c492020 Fix #3396: Pixiv novel cover IDs are parsed incorrectly. 2017-11-21 13:03:03 -06:00
r888888888
502f1298a9 fix unit tests 2017-11-20 16:30:07 -08:00
evazion
22980c845a tests: fix pixiv ugoira urls. 2017-07-25 00:36:18 -05:00
evazion
5684ba3b1d tests: factor out assert_downloaded to DownloadTestHelper. 2017-06-21 21:10:07 -05:00
evazion
b68cb174f5 tests: fix pixiv tests to use https://i.pixiv.net. 2017-04-15 23:14:41 -05:00
r888888888
0b8d4105aa fix tests 2017-04-04 12:39:17 -07:00
evazion
beddc34958 tests: set Delayed::Worker.delay_jobs = false globally. 2017-02-04 17:18:51 -06:00
Albert Yi
0ea7d78584 remove usage of vcr cassettes; delete unused fixtures; fix some broken unit tests 2016-12-28 15:47:28 -08:00
evazion
e7bc3048ed Test downloading pixiv profile/background/novel images. 2016-12-04 02:04:44 -06:00
r888888888
fc7afd44ea refactor source pixiv test
refactor pixiv download tests
refactor upload test
refactor nico seiga test
refactor twitter tests
2016-09-28 11:25:29 -07:00
r888888888
58aa5c6d66 fix tests 2016-05-28 14:08:44 -07:00
r888888888
66dd4f072e update tests 2015-06-02 19:20:48 -07:00
r888888888
ceaa98b105 fix tests 2015-05-21 12:11:11 -07:00
r888888888
9bdc0116bf fix pixiv tests 2015-05-14 16:50:24 -07:00
r888888888
8d4c9d7955 fix pixiv tests 2014-10-22 17:22:36 -07:00
r888888888
3bb06c2be4 integrate ugoiras into zip+webm+preview 2014-10-19 02:30:02 -07:00
evazion
339e88584d Rewrite tests for downloading new Pixiv URLs.
* Test that downloading the HTML page downloads the full size image
  instead.
* Test that downloading a small or medium size image downloads the
  full size instead.
* Test the above for both single illustrations and for manga galleries,
  for new and old posts (i.e. uploaded before the Pixiv URL changes versus
  after), and for different file extensions.
* Test trying to download ugoira zip files. These tests are expected to
  fail because we don't support ugoira yet.
* Also add some more tests for fetching source data.
2014-10-04 12:45:37 -05:00
Toks
940909ca9c #1866: Support pixiv small "_s" thumbnail source rewriting 2013-11-26 14:36:39 -05:00
r888888888
850270dafa add vcr cassettes for pixiv tests 2013-04-30 17:35:24 -07:00
r888888888
6caa0f0436 fix tests 2013-04-14 22:24:55 -07:00
小太
cba839ba76 Kill trailing whitespace in ruby files 2013-03-19 23:10:10 +11:00
albert
365e04bb40 fixes #194 2011-12-30 16:30:32 -05:00
albert
aa2b65c48b tinami/pixa downloads now supported 2011-09-30 14:27:04 -04:00