jaculabilis/danbooru
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
6,757 Commits 1 Branch 0 Tags
8a012d4c91e47e4426ded1ffa97935202cfd2afa
Commit Graph

108 Commits

Author SHA1 Message Date
evazion
5ad06a4034 Fix #3552: Upload filesize limits can be bypassed. 
* Change `http_get_streaming` to write the output file directly,
  instead of taking a callback.

* Track the filesize as the download progresses and abort when it
  exceeds the limit.

* Don't save the Content-Type (it's not used anywhere).
 2018-02-27 18:22:34 -06:00
evazion
c9eee7e4d4 Fix #3528: Prevent CloudFlare from altering images. 2018-02-24 13:42:00 -06:00
evazion
e48b75c261 downloads: rewrite url in download!, not http_get_streaming. 
Refactor Downloads::File#http_get_streaming to just download the given
url, not rewrite it.

Don't clobber @source or @data in `#size` either.
 2018-02-24 11:23:27 -06:00
Albert Yi
d8340f83db Revert "Fix #3528: Add method to prevent image-hosting CloudFlare sites from altering the image" 2018-02-21 17:20:13 -08:00
evazion
263fd0eef1 Fix #3528: Prevent CloudFlare from altering images. 2018-02-17 11:43:27 -06:00
evazion
1eff41a6f9 downloads: rewrite url in download!, not http_get_streaming. 
Refactor Downloads::File#http_get_streaming to just download the given
url, not rewrite it.

Don't clobber @source or @data in `#size` either.
 2018-02-17 11:11:50 -06:00
evazion
0b5ed163fe Fix #3395: fix pixiv API exception when uploading bad pixiv id images. 
Fixes an exception when attempting to upload a Pixiv image from a
deleted work. The download strategy tries to fetch the source data in
the course of rewriting the URL, which fails if the work has been
deleted from Pixiv.

Raise a BadIDError and leave the URL as-is (don't rewrite it).
 2017-11-21 14:40:39 -06:00
evazion
fa22e419af Fix #3385: Bookmarklet sets wrong source for Tumblr posts 
Due to missing || operators, for Pawoo/Tumblr/Artstation uploads the
source wasn't changed from the direct image URL to the html page URL
after upload.
 2017-11-18 15:44:16 -06:00
r888888888
3c524aa5e6 fix bug with source::site 2017-11-17 17:26:33 -08:00
evazion
eeb41d2ffd twitter: get status id from referer url if not in direct url. 2017-11-16 13:29:58 -06:00
r888888888
c2b49bf2b7 fixes #3293 2017-09-13 11:14:35 -07:00
evazion
7ae860a8b8 Drop Pixiv whitecube support (#2681, #3206). 2017-07-26 18:06:46 -05:00
evazion
96fddc5bec Fix #3234: Incorrect filesize on upload page. 2017-07-22 00:34:30 -05:00
evazion
8125d8e796 Set the default User-Agent sent for outgoing http requests. 
* Add a `Danbooru.config.http_headers` config option.

* Refactor various places to use this option instead of setting the user
  agent manually.
 2017-07-19 20:25:27 -05:00
r888888888
ee6581ab7f restore streaming behavior for Downloads::File#http_get_streaming 2017-07-13 15:31:29 -07:00
evazion
5066f75d8c Fix Net::HTTPOK#read_body called twice error during upload (#3135) 
Fixes this exception:

    An error occurred: error: IOError - Net::HTTPOK#read_body called twice

    /home/admin/.rbenv/versions/2.3.4/lib/ruby/2.3.0/net/http/response.rb:196:in `read_body'
    /home/admin/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/httparty-0.15.5/lib/httparty/response.rb:91:in `method_missing'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:43:in `block (2 levels) in download!'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:41:in `open'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:41:in `block in download!'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:100:in `http_get_streaming'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:39:in `download!'
    /home/admin/src/danbooru/app/models/upload.rb:408:in `download_from_source'
    /home/admin/src/danbooru/app/models/upload.rb:113:in `block in process_upload'
 2017-07-13 15:41:36 -05:00
r888888888
97ce9cafac add httparty option config 2017-07-12 15:52:48 -07:00
r888888888
eb6c5e3af5 switch to httparty 2017-07-12 15:52:48 -07:00
evazion
c3ac4f3329 Fix #3197 - Post Replacement comment: use Replacement URL instead of Final Source 
Record the URL of the image that was actually downloaded as the replacement URL.
 2017-07-02 21:00:26 -05:00
evazion
030052bf14 tumblr: rewrite html pages to image url. 2017-06-25 15:34:15 -05:00
evazion
118843b26a tumblr: fetch tags, commentary, and image urls. 2017-06-25 15:34:15 -05:00
evazion
421bbc35a2 tumblr: rewrite samples to biggest available version. 2017-06-21 22:30:53 -05:00
evazion
a9b3d28215 tumblr: fix edgecastcdn.net rewrite + add test. 
https://data.tumblr.com doesn't have a valid SSL cert, so we must use
http:// instead when we rewrite to it.
 2017-06-21 21:10:07 -05:00
evazion
655e53fe65 pawoo: rewrite /small/ images to /original/ 2017-06-19 17:10:02 -05:00
evazion
215569fbe0 artstation: fix source to page url after download. 2017-06-07 10:20:17 -05:00
evazion
b3e533f3ff Add download rewrite for https://$artist.artstation.com/projects/$id. 2017-06-05 14:03:51 -05:00
evazion
b2f62482a1 Remove artist commentary handling from download rewrite strategies. 2017-05-10 18:01:51 -05:00
evazion
db1fa60a43 Set pawoo sources to html page after upload. 2017-05-03 00:09:51 -05:00
r888888888
bd7f4820d7 fix source js for pawoo sites 2017-04-20 16:43:41 -07:00
r888888888
b053a2d783 add source + rewrite strategy for pawoo 2017-04-20 15:50:22 -07:00
evazion
af4708dde9 Fix typo in i.pximg.net download strategy. 2017-04-03 17:17:22 -05:00
r888888888
8d58e27d0b additional support for pximg.net hosts 2017-04-03 15:03:26 -07:00
r888888888
7a3d1c7d1d dont download original artstation images if they dont exist 2017-02-07 13:40:19 -08:00
r888888888
9996030496 include default referer url for art station strategy 2017-02-06 17:37:45 -08:00
r888888888
2e69cd619e resize medium + small artstation urls also 2017-02-06 15:22:06 -08:00
r888888888
5c16e9ce48 add download strategy for artstation 2017-02-06 12:03:00 -08:00
Albert Yi
b0f2b1e8dd Merge pull request #2782 from evazion/fix-2779 
Fix #2779: Erroneous pixiv_id parsed for novel/background/profile images
 2016-12-05 11:44:38 -08:00
Albert Yi
bd9939a50b some additional documentation about rewrite strategies 2016-12-05 11:43:18 -08:00
evazion
838d6f6eea Fix #2460: Uploading pixiv user's background image throws error. 
Fix exception when uploading these types of URLs:

* http://img12.pixiv.net/profile/rapattu/119950.jpg
* http://i2.pixiv.net/img130/profile/minono_aki/8733472.jpg
* http://i2.pixiv.net/img20/img/a-park/novel/3607898.jpg
* http://i1.pixiv.net/novel-cover-original/img/2016/11/03/20/10/58/7436075_f75af69f3eacd1656d3733c72aa959cf.jpg
* http://i1.pixiv.net/background/img/2016/05/17/12/05/48/2074388_d4ac52034f7ca0af3e083d59fde7e97f.jpg

Skip rewriting sources that don't have illust IDs to avoid this
exception. Also fix `tags` so that it doesn't blow up when fetching
source data for these types of sources.
 2016-12-04 02:04:44 -06:00
evazion
3eafc5ae6a Don't clobber image board sources. 2016-11-19 02:29:33 -06:00
evazion
f94dfbf17e Fix bookmarklet for whitecube URLs. 
Make the bookmarklet work for URLs like this:

* http://www.pixiv.net/whitecube/user/607414/illust/59510800
* http://www.pixiv.net/whitecube/user/772090/illust/59595705
 2016-10-22 16:11:44 -05:00
r888888888
4fa2741ed2 include whitecube source/rewrite strategies 2016-09-28 11:24:54 -07:00
Toks
98ef1816ba Rewrite nijie html view page to image url 2015-12-22 12:29:46 -05:00
Toks
aa212e3616 Fix error when uploading old deviantart url format 2015-09-21 10:47:24 -04:00
Toks
2359f41441 Reorganize rewrite strats slightly 2015-09-19 10:26:40 -04:00
Toks
a072fae19d Fix "EOFError - end of file reached" for https tumblr uploads 2015-09-09 10:41:59 -04:00
Toks
12be880fa9 Support fetching artist commentary from deviantart 2015-09-03 20:03:56 -04:00
r888888888
f85d5a5570 addresses #2498: revised implementation 2015-08-18 12:45:10 -07:00
r888888888
407523b04c addresses #2498: Protect source downloader against server-side request forgery attacks 2015-08-18 12:26:26 -07:00
Toks
e165cb0ab2 fixes #2491 2015-08-15 17:12:00 -04:00