jaculabilis/danbooru
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
8,092 Commits 1 Branch 0 Tags
be36968b6d8ae4b2c8d2b2dfd544eada20046bea
Commit Graph

125 Commits

Author SHA1 Message Date
evazion
7524d52276 Fix #3985: Uploads: 405 Method Not Allowed. 2018-11-11 20:18:21 -06:00
r888888888
192ac128cd add additional meta tags for seo 2018-10-20 17:15:29 -07:00
evazion
f4e08ef30d Downloads::File: fix SSRF inside is_cloudflare? (#2498). 
Fixes the banned IP check not being applied when sending the HEAD
request for is_cloudflare?.

Also fixes the `#size` method not using the uncached url (which meant
the bookmarklet could report the wrong filesize on artstation uploads).
 2018-09-19 20:11:53 -05:00
evazion
2f17082e73 Downloads::File: fix SSRF when following redirects (#2498). 
Fixes the banned IP check not being applied when following redirects:

  http://danbooru.donmai.us/uploads/new?url=http://httpbin.org/redirect-to%3Furl=http://127.0.0.1/test.jpg
 2018-09-18 16:00:03 -05:00
evazion
99221e4028 Downloads::File: fix SSRF attack when fetching remote size (#2498). 
Fixes the banned IP check not being applied when fetching the remote
file size. This allowed one to trick Danbooru into sending HEAD requests
to private IPs:

  http://danbooru.donmai.us/uploads/new?url=http://127.0.0.1/test.jpg
 2018-09-18 12:16:27 -05:00
evazion
488b1fd994 Downloads::File: memoize strategy. 
Also remove unused options param from constructor
 2018-09-18 10:22:15 -05:00
evazion
9cdfbba6c2 Fix #3910: Corrupted images during upload. 
Use a fresh tempfile for each download attempt instead of reusing the same
file (and having to rewind/truncate it after each failed attempt).
 2018-09-18 10:01:44 -05:00
evazion
d3c135ec72 Downloads::File#http_get_streaming: clean up retry logic. 
Replace handrolled retry logic with retriable gem (already pulled in by another gem).
 2018-09-18 09:44:15 -05:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
Albert Yi
e620abad08 add checks for width=640 and width=500h in tumblr rewriter (#3732) 2018-07-24 16:42:30 -07:00
Albert Yi
f6440ca70d fix upload preprocessing for pixiv posts 
fixes #3782
 2018-07-23 16:24:31 -07:00
Albert Yi
f9b63702bf support artist-path urls in deviant art rewrite strategy (#3771) 2018-07-09 16:47:59 -07:00
Albert Yi
01250faaac temporary fix for nil urls 2018-05-25 09:39:59 -07:00
Albert Yi
23bef55632 Merge pull request #3700 from evazion/fix-3659 
Fix uploads getting stuck in 'processing' state (fix #3659)
 2018-05-07 17:36:03 -07:00
evazion
181a906766 Fix #3695: Bookmarklet breaks on pixiv fanbox direct image links. 2018-05-05 12:21:11 -05:00
evazion
c584ca5b19 Fix uploads getting stuck in 'processing' state (fix #3659). 
Bug: if an upload timed out while downloading the file, Upload#process!
would catch the error and attempt to retry, but since the upload was
already in the 'processing' state, on the second try `process!` would
bail out immediately and leave the upload stuck in the 'processing' state.

Fix: remove the retry logic from Upload#process!. Let Downloads::File#download!
(which had its own retry logic) handle it instead.
 2018-05-05 11:42:40 -05:00
evazion
c76463f34d uploads: use storage manager to distribute files. 
Refactors the upload process to pass around temp files, rather than
passing around file paths and directly writing output to the local
filesystem. This way we can pass the storage manager the preview /
sample / original temp files, so it can deal with storage itself.

* Change Download::File#download! to return a temp file.

* Change DanbooruImageResizer and PixivUgoiraConverter to accept/return
  temp files instead of file paths.

* Change Upload#generate_resizes to return temp files for previews and samples.

* Change Upload#generate_resizes to generate ugoira .webm samples
  synchronously instead of asynchronously.
 2018-03-20 19:49:06 -05:00
evazion
5ad06a4034 Fix #3552: Upload filesize limits can be bypassed. 
* Change `http_get_streaming` to write the output file directly,
  instead of taking a callback.

* Track the filesize as the download progresses and abort when it
  exceeds the limit.

* Don't save the Content-Type (it's not used anywhere).
 2018-02-27 18:22:34 -06:00
evazion
c9eee7e4d4 Fix #3528: Prevent CloudFlare from altering images. 2018-02-24 13:42:00 -06:00
evazion
e48b75c261 downloads: rewrite url in download!, not http_get_streaming. 
Refactor Downloads::File#http_get_streaming to just download the given
url, not rewrite it.

Don't clobber @source or @data in `#size` either.
 2018-02-24 11:23:27 -06:00
Albert Yi
d8340f83db Revert "Fix #3528: Add method to prevent image-hosting CloudFlare sites from altering the image" 2018-02-21 17:20:13 -08:00
evazion
263fd0eef1 Fix #3528: Prevent CloudFlare from altering images. 2018-02-17 11:43:27 -06:00
evazion
1eff41a6f9 downloads: rewrite url in download!, not http_get_streaming. 
Refactor Downloads::File#http_get_streaming to just download the given
url, not rewrite it.

Don't clobber @source or @data in `#size` either.
 2018-02-17 11:11:50 -06:00
evazion
0b5ed163fe Fix #3395: fix pixiv API exception when uploading bad pixiv id images. 
Fixes an exception when attempting to upload a Pixiv image from a
deleted work. The download strategy tries to fetch the source data in
the course of rewriting the URL, which fails if the work has been
deleted from Pixiv.

Raise a BadIDError and leave the URL as-is (don't rewrite it).
 2017-11-21 14:40:39 -06:00
evazion
fa22e419af Fix #3385: Bookmarklet sets wrong source for Tumblr posts 
Due to missing || operators, for Pawoo/Tumblr/Artstation uploads the
source wasn't changed from the direct image URL to the html page URL
after upload.
 2017-11-18 15:44:16 -06:00
r888888888
3c524aa5e6 fix bug with source::site 2017-11-17 17:26:33 -08:00
evazion
eeb41d2ffd twitter: get status id from referer url if not in direct url. 2017-11-16 13:29:58 -06:00
r888888888
c2b49bf2b7 fixes #3293 2017-09-13 11:14:35 -07:00
evazion
7ae860a8b8 Drop Pixiv whitecube support (#2681, #3206). 2017-07-26 18:06:46 -05:00
evazion
96fddc5bec Fix #3234: Incorrect filesize on upload page. 2017-07-22 00:34:30 -05:00
evazion
8125d8e796 Set the default User-Agent sent for outgoing http requests. 
* Add a `Danbooru.config.http_headers` config option.

* Refactor various places to use this option instead of setting the user
  agent manually.
 2017-07-19 20:25:27 -05:00
r888888888
ee6581ab7f restore streaming behavior for Downloads::File#http_get_streaming 2017-07-13 15:31:29 -07:00
evazion
5066f75d8c Fix Net::HTTPOK#read_body called twice error during upload (#3135) 
Fixes this exception:

    An error occurred: error: IOError - Net::HTTPOK#read_body called twice

    /home/admin/.rbenv/versions/2.3.4/lib/ruby/2.3.0/net/http/response.rb:196:in `read_body'
    /home/admin/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/httparty-0.15.5/lib/httparty/response.rb:91:in `method_missing'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:43:in `block (2 levels) in download!'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:41:in `open'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:41:in `block in download!'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:100:in `http_get_streaming'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:39:in `download!'
    /home/admin/src/danbooru/app/models/upload.rb:408:in `download_from_source'
    /home/admin/src/danbooru/app/models/upload.rb:113:in `block in process_upload'
 2017-07-13 15:41:36 -05:00
r888888888
97ce9cafac add httparty option config 2017-07-12 15:52:48 -07:00
r888888888
eb6c5e3af5 switch to httparty 2017-07-12 15:52:48 -07:00
evazion
c3ac4f3329 Fix #3197 - Post Replacement comment: use Replacement URL instead of Final Source 
Record the URL of the image that was actually downloaded as the replacement URL.
 2017-07-02 21:00:26 -05:00
evazion
030052bf14 tumblr: rewrite html pages to image url. 2017-06-25 15:34:15 -05:00
evazion
118843b26a tumblr: fetch tags, commentary, and image urls. 2017-06-25 15:34:15 -05:00
evazion
421bbc35a2 tumblr: rewrite samples to biggest available version. 2017-06-21 22:30:53 -05:00
evazion
a9b3d28215 tumblr: fix edgecastcdn.net rewrite + add test. 
https://data.tumblr.com doesn't have a valid SSL cert, so we must use
http:// instead when we rewrite to it.
 2017-06-21 21:10:07 -05:00
evazion
655e53fe65 pawoo: rewrite /small/ images to /original/ 2017-06-19 17:10:02 -05:00
evazion
215569fbe0 artstation: fix source to page url after download. 2017-06-07 10:20:17 -05:00
evazion
b3e533f3ff Add download rewrite for https://$artist.artstation.com/projects/$id. 2017-06-05 14:03:51 -05:00
evazion
b2f62482a1 Remove artist commentary handling from download rewrite strategies. 2017-05-10 18:01:51 -05:00
evazion
db1fa60a43 Set pawoo sources to html page after upload. 2017-05-03 00:09:51 -05:00
r888888888
bd7f4820d7 fix source js for pawoo sites 2017-04-20 16:43:41 -07:00
r888888888
b053a2d783 add source + rewrite strategy for pawoo 2017-04-20 15:50:22 -07:00
evazion
af4708dde9 Fix typo in i.pximg.net download strategy. 2017-04-03 17:17:22 -05:00
r888888888
8d58e27d0b additional support for pximg.net hosts 2017-04-03 15:03:26 -07:00
r888888888
7a3d1c7d1d dont download original artstation images if they dont exist 2017-02-07 13:40:19 -08:00