Commit Graph

78 Commits

Author SHA1 Message Date
evazion
eba6440b8b Fix #4144: Deviantart Eclipse update broke strategy. 2019-08-28 23:40:29 -05:00
Albert Yi
6612aa1af9 fix unit tests 2019-04-25 18:13:59 -07:00
evazion
7524d52276 Fix #3985: Uploads: 405 Method Not Allowed. 2018-11-11 20:18:21 -06:00
evazion
5525bbe1ca nijie: normalize all thumbnail urls (#3919). 2018-09-23 20:08:14 -05:00
evazion
f4e08ef30d Downloads::File: fix SSRF inside is_cloudflare? (#2498).
Fixes the banned IP check not being applied when sending the HEAD
request for is_cloudflare?.

Also fixes the `#size` method not using the uncached url (which meant
the bookmarklet could report the wrong filesize on artstation uploads).
2018-09-19 20:11:53 -05:00
evazion
2f17082e73 Downloads::File: fix SSRF when following redirects (#2498).
Fixes the banned IP check not being applied when following redirects:

  http://danbooru.donmai.us/uploads/new?url=http://httpbin.org/redirect-to%3Furl=http://127.0.0.1/test.jpg
2018-09-18 16:00:03 -05:00
evazion
99221e4028 Downloads::File: fix SSRF attack when fetching remote size (#2498).
Fixes the banned IP check not being applied when fetching the remote
file size. This allowed one to trick Danbooru into sending HEAD requests
to private IPs:

  http://danbooru.donmai.us/uploads/new?url=http://127.0.0.1/test.jpg
2018-09-18 12:16:27 -05:00
evazion
9cdfbba6c2 Fix #3910: Corrupted images during upload.
Use a fresh tempfile for each download attempt instead of reusing the same
file (and having to rewind/truncate it after each failed attempt).
2018-09-18 10:01:44 -05:00
evazion
d3c135ec72 Downloads::File#http_get_streaming: clean up retry logic.
Replace handrolled retry logic with retriable gem (already pulled in by another gem).
2018-09-18 09:44:15 -05:00
evazion
96e89cecfb tests: move twitter canonical url test. 2018-09-17 23:27:53 -05:00
evazion
cae78fa8ee moebooru: move tests from unit/downloads to unit/sources. 2018-09-16 21:00:11 -05:00
Albert Yi
266c7c0d5b cache api clients 2018-09-11 14:19:17 -07:00
evazion
0d7b7e6e85 twitter tests: fix dead tweet. 2018-09-09 14:10:43 -05:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
Albert Yi
77854349e5 testing 2018-07-26 18:11:19 -07:00
Albert Yi
5ae37597cd fixes #3728 2018-05-25 13:24:49 -07:00
Albert Yi
6839249e9d add logic for persisting pixiv sessions in tests 2018-05-10 13:59:57 -07:00
Albert Yi
5bca31bad1 add retry monkey patch for mechanize 2018-05-09 16:45:03 -07:00
Albert Yi
e23814be92 fix pixiv agent shutdown 2018-05-09 16:13:47 -07:00
Albert Yi
d4a2521eec reset connection for pixiv tests 2018-05-09 14:59:18 -07:00
Albert Yi
99012ff342 fix tests 2018-05-09 11:59:51 -07:00
evazion
181a906766 Fix #3695: Bookmarklet breaks on pixiv fanbox direct image links. 2018-05-05 12:21:11 -05:00
r888888888
abce4d2551 Raise error on unpermitted params.
Fail loudly if we forget to whitelist a param instead of silently
ignoring it.

misc models: convert to strong params.

artist commentaries: convert to strong params.

* Disallow changing or setting post_id to a nonexistent post.

artists: convert to strong params.

* Disallow setting `is_banned` in create/update actions. Changing it
  this way instead of with the ban/unban actions would leave the artist in
  a partially banned state.

bans: convert to strong params.

* Disallow changing the user_id after the ban has been created.

comments: convert to strong params.

favorite groups: convert to strong params.

news updates: convert to strong params.

post appeals: convert to strong params.

post flags: convert to strong params.

* Disallow users from setting the `is_deleted` / `is_resolved` flags.

ip bans: convert to strong params.

user feedbacks: convert to strong params.

* Disallow users from setting `disable_dmail_notification` when creating feedbacks.
* Disallow changing the user_id after the feedback has been created.

notes: convert to strong params.

wiki pages: convert to strong params.

* Also fix non-Builders being able to delete wiki pages.

saved searches: convert to strong params.

pools: convert to strong params.

* Disallow setting `post_count` or `is_deleted` in create/update actions.

janitor trials: convert to strong params.

post disapprovals: convert to strong params.

* Factor out quick-mod bar to shared partial.
* Fix quick-mod bar to use `Post#is_approvable?` to determine visibility
  of Approve button.

dmail filters: convert to strong params.

password resets: convert to strong params.

user name change requests: convert to strong params.

posts: convert to strong params.

users: convert to strong params.

* Disallow setting password_hash, last_logged_in_at, last_forum_read_at,
  has_mail, and dmail_filter_attributes[user_id].

* Remove initialize_default_image_size (dead code).

uploads: convert to strong params.

* Remove `initialize_status` because status already defaults to pending
  in the database.

tag aliases/implications: convert to strong params.

tags: convert to strong params.

forum posts: convert to strong params.

* Disallow changing the topic_id after creating the post.
* Disallow setting is_deleted (destroy/undelete actions should be used instead).
* Remove is_sticky / is_locked (nonexistent attributes).

forum topics: convert to strong params.

* merges https://github.com/evazion/danbooru/tree/wip-rails-5.1
* lock pg gem to 0.21 (1.0.0 is incompatible with rails 5.1.4)
* switch to factorybot and change all references

Co-authored-by: r888888888 <r888888888@gmail.com>
Co-authored-by: evazion <noizave@gmail.com>

add diffs
2018-04-06 18:09:57 -07:00
evazion
d089be9f8a tests: fix upload tests. 2018-03-20 19:49:58 -05:00
evazion
5ad06a4034 Fix #3552: Upload filesize limits can be bypassed.
* Change `http_get_streaming` to write the output file directly,
  instead of taking a callback.

* Track the filesize as the download progresses and abort when it
  exceeds the limit.

* Don't save the Content-Type (it's not used anywhere).
2018-02-27 18:22:34 -06:00
evazion
c9eee7e4d4 Fix #3528: Prevent CloudFlare from altering images. 2018-02-24 13:42:00 -06:00
evazion
b859a1f714 downloads: add tests for untested sites. 2018-02-24 13:42:00 -06:00
Albert Yi
d8340f83db Revert "Fix #3528: Add method to prevent image-hosting CloudFlare sites from altering the image" 2018-02-21 17:20:13 -08:00
evazion
263fd0eef1 Fix #3528: Prevent CloudFlare from altering images. 2018-02-17 11:43:27 -06:00
evazion
a923a64220 tests: fix pixiv profile image test. 2018-01-20 14:51:05 -06:00
evazion
53f7207716 tests: remove bad_id pixiv test. 2018-01-20 14:50:44 -06:00
evazion
0b5ed163fe Fix #3395: fix pixiv API exception when uploading bad pixiv id images.
Fixes an exception when attempting to upload a Pixiv image from a
deleted work. The download strategy tries to fetch the source data in
the course of rewriting the URL, which fails if the work has been
deleted from Pixiv.

Raise a BadIDError and leave the URL as-is (don't rewrite it).
2017-11-21 14:40:39 -06:00
evazion
5ba63733d3 download/pixiv_test.rb: fix broken pixiv download tests.
This post was deleted:

  http://www.pixiv.net/member_illust.php?mode=medium&illust_id=46304614

Switch to this instead:

  http://www.pixiv.net/member_illust.php?mode=medium&illust_id=46324488
2017-11-21 13:48:07 -06:00
evazion
a14c492020 Fix #3396: Pixiv novel cover IDs are parsed incorrectly. 2017-11-21 13:03:03 -06:00
r888888888
502f1298a9 fix unit tests 2017-11-20 16:30:07 -08:00
r888888888
3c524aa5e6 fix bug with source::site 2017-11-17 17:26:33 -08:00
r888888888
c2b49bf2b7 fixes #3293 2017-09-13 11:14:35 -07:00
evazion
22980c845a tests: fix pixiv ugoira urls. 2017-07-25 00:36:18 -05:00
r888888888
ee6581ab7f restore streaming behavior for Downloads::File#http_get_streaming 2017-07-13 15:31:29 -07:00
evazion
030052bf14 tumblr: rewrite html pages to image url. 2017-06-25 15:34:15 -05:00
evazion
421bbc35a2 tumblr: rewrite samples to biggest available version. 2017-06-21 22:30:53 -05:00
evazion
a9b3d28215 tumblr: fix edgecastcdn.net rewrite + add test.
https://data.tumblr.com doesn't have a valid SSL cert, so we must use
http:// instead when we rewrite to it.
2017-06-21 21:10:07 -05:00
evazion
16a51533bc tests: refactor tumblr tests to use assert_downloaded. 2017-06-21 21:10:07 -05:00
evazion
5684ba3b1d tests: factor out assert_downloaded to DownloadTestHelper. 2017-06-21 21:10:07 -05:00
evazion
b3e533f3ff Add download rewrite for https://$artist.artstation.com/projects/$id. 2017-06-05 14:03:51 -05:00
evazion
b68cb174f5 tests: fix pixiv tests to use https://i.pixiv.net. 2017-04-15 23:14:41 -05:00
r888888888
0b8d4105aa fix tests 2017-04-04 12:39:17 -07:00
r888888888
7a3d1c7d1d dont download original artstation images if they dont exist 2017-02-07 13:40:19 -08:00
r888888888
5c16e9ce48 add download strategy for artstation 2017-02-06 12:03:00 -08:00
evazion
beddc34958 tests: set Delayed::Worker.delay_jobs = false globally. 2017-02-04 17:18:51 -06:00