jaculabilis/danbooru
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
8,670 Commits 1 Branch 0 Tags
153a8339ab4505a36e44b28e8746d4d3a514fb8f
Commit Graph

60 Commits

Author SHA1 Message Date
evazion
153a8339ab Inherit errors from StandardError instead of Exception. 2020-01-11 19:07:28 -06:00
evazion
309821bf73 rubocop: fix various style issues. 2019-12-22 21:23:37 -06:00
evazion
3cc636e719 Fix uninitialized constant error with Resolv class. 
Fix uninitialized constant error (introduced by the switch to zeitwerk?)
 2019-12-13 18:19:41 -06:00
evazion
a6efaa54a1 Downloads::File: fix following 302 redirects during download. 
Fixes downloading yande.re preview images for iqdb. Yande.re previews
return a 302 redirect to the actual file. Before the html in the body of
the 302 response would get prepended to the image file.

https://files.yande.re/data/preview/12/8f/128fb480d8589be26d1dce7e1d841bcb.jpg
=> https://assets.yande.re/data/preview/12/8f/128fb480d8589be26d1dce7e1d841bcb.jpg
 2019-10-26 14:04:19 -05:00
evazion
7524d52276 Fix #3985: Uploads: 405 Method Not Allowed. 2018-11-11 20:18:21 -06:00
r888888888
192ac128cd add additional meta tags for seo 2018-10-20 17:15:29 -07:00
evazion
f4e08ef30d Downloads::File: fix SSRF inside is_cloudflare? (#2498). 
Fixes the banned IP check not being applied when sending the HEAD
request for is_cloudflare?.

Also fixes the `#size` method not using the uncached url (which meant
the bookmarklet could report the wrong filesize on artstation uploads).
 2018-09-19 20:11:53 -05:00
evazion
2f17082e73 Downloads::File: fix SSRF when following redirects (#2498). 
Fixes the banned IP check not being applied when following redirects:

  http://danbooru.donmai.us/uploads/new?url=http://httpbin.org/redirect-to%3Furl=http://127.0.0.1/test.jpg
 2018-09-18 16:00:03 -05:00
evazion
99221e4028 Downloads::File: fix SSRF attack when fetching remote size (#2498). 
Fixes the banned IP check not being applied when fetching the remote
file size. This allowed one to trick Danbooru into sending HEAD requests
to private IPs:

  http://danbooru.donmai.us/uploads/new?url=http://127.0.0.1/test.jpg
 2018-09-18 12:16:27 -05:00
evazion
488b1fd994 Downloads::File: memoize strategy. 
Also remove unused options param from constructor
 2018-09-18 10:22:15 -05:00
evazion
9cdfbba6c2 Fix #3910: Corrupted images during upload. 
Use a fresh tempfile for each download attempt instead of reusing the same
file (and having to rewind/truncate it after each failed attempt).
 2018-09-18 10:01:44 -05:00
evazion
d3c135ec72 Downloads::File#http_get_streaming: clean up retry logic. 
Replace handrolled retry logic with retriable gem (already pulled in by another gem).
 2018-09-18 09:44:15 -05:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
Albert Yi
f6440ca70d fix upload preprocessing for pixiv posts 
fixes #3782
 2018-07-23 16:24:31 -07:00
Albert Yi
01250faaac temporary fix for nil urls 2018-05-25 09:39:59 -07:00
evazion
c584ca5b19 Fix uploads getting stuck in 'processing' state (fix #3659). 
Bug: if an upload timed out while downloading the file, Upload#process!
would catch the error and attempt to retry, but since the upload was
already in the 'processing' state, on the second try `process!` would
bail out immediately and leave the upload stuck in the 'processing' state.

Fix: remove the retry logic from Upload#process!. Let Downloads::File#download!
(which had its own retry logic) handle it instead.
 2018-05-05 11:42:40 -05:00
evazion
c76463f34d uploads: use storage manager to distribute files. 
Refactors the upload process to pass around temp files, rather than
passing around file paths and directly writing output to the local
filesystem. This way we can pass the storage manager the preview /
sample / original temp files, so it can deal with storage itself.

* Change Download::File#download! to return a temp file.

* Change DanbooruImageResizer and PixivUgoiraConverter to accept/return
  temp files instead of file paths.

* Change Upload#generate_resizes to return temp files for previews and samples.

* Change Upload#generate_resizes to generate ugoira .webm samples
  synchronously instead of asynchronously.
 2018-03-20 19:49:06 -05:00
evazion
5ad06a4034 Fix #3552: Upload filesize limits can be bypassed. 
* Change `http_get_streaming` to write the output file directly,
  instead of taking a callback.

* Track the filesize as the download progresses and abort when it
  exceeds the limit.

* Don't save the Content-Type (it's not used anywhere).
 2018-02-27 18:22:34 -06:00
evazion
c9eee7e4d4 Fix #3528: Prevent CloudFlare from altering images. 2018-02-24 13:42:00 -06:00
evazion
e48b75c261 downloads: rewrite url in download!, not http_get_streaming. 
Refactor Downloads::File#http_get_streaming to just download the given
url, not rewrite it.

Don't clobber @source or @data in `#size` either.
 2018-02-24 11:23:27 -06:00
Albert Yi
d8340f83db Revert "Fix #3528: Add method to prevent image-hosting CloudFlare sites from altering the image" 2018-02-21 17:20:13 -08:00
evazion
263fd0eef1 Fix #3528: Prevent CloudFlare from altering images. 2018-02-17 11:43:27 -06:00
evazion
1eff41a6f9 downloads: rewrite url in download!, not http_get_streaming. 
Refactor Downloads::File#http_get_streaming to just download the given
url, not rewrite it.

Don't clobber @source or @data in `#size` either.
 2018-02-17 11:11:50 -06:00
evazion
fa22e419af Fix #3385: Bookmarklet sets wrong source for Tumblr posts 
Due to missing || operators, for Pawoo/Tumblr/Artstation uploads the
source wasn't changed from the direct image URL to the html page URL
after upload.
 2017-11-18 15:44:16 -06:00
evazion
eeb41d2ffd twitter: get status id from referer url if not in direct url. 2017-11-16 13:29:58 -06:00
r888888888
c2b49bf2b7 fixes #3293 2017-09-13 11:14:35 -07:00
evazion
96fddc5bec Fix #3234: Incorrect filesize on upload page. 2017-07-22 00:34:30 -05:00
evazion
8125d8e796 Set the default User-Agent sent for outgoing http requests. 
* Add a `Danbooru.config.http_headers` config option.

* Refactor various places to use this option instead of setting the user
  agent manually.
 2017-07-19 20:25:27 -05:00
r888888888
ee6581ab7f restore streaming behavior for Downloads::File#http_get_streaming 2017-07-13 15:31:29 -07:00
evazion
5066f75d8c Fix Net::HTTPOK#read_body called twice error during upload (#3135) 
Fixes this exception:

    An error occurred: error: IOError - Net::HTTPOK#read_body called twice

    /home/admin/.rbenv/versions/2.3.4/lib/ruby/2.3.0/net/http/response.rb:196:in `read_body'
    /home/admin/.rbenv/versions/2.3.4/lib/ruby/gems/2.3.0/gems/httparty-0.15.5/lib/httparty/response.rb:91:in `method_missing'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:43:in `block (2 levels) in download!'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:41:in `open'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:41:in `block in download!'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:100:in `http_get_streaming'
    /home/admin/src/danbooru/app/logical/downloads/file.rb:39:in `download!'
    /home/admin/src/danbooru/app/models/upload.rb:408:in `download_from_source'
    /home/admin/src/danbooru/app/models/upload.rb:113:in `block in process_upload'
 2017-07-13 15:41:36 -05:00
r888888888
97ce9cafac add httparty option config 2017-07-12 15:52:48 -07:00
r888888888
eb6c5e3af5 switch to httparty 2017-07-12 15:52:48 -07:00
evazion
c3ac4f3329 Fix #3197 - Post Replacement comment: use Replacement URL instead of Final Source 
Record the URL of the image that was actually downloaded as the replacement URL.
 2017-07-02 21:00:26 -05:00
evazion
118843b26a tumblr: fetch tags, commentary, and image urls. 2017-06-25 15:34:15 -05:00
evazion
215569fbe0 artstation: fix source to page url after download. 2017-06-07 10:20:17 -05:00
evazion
db1fa60a43 Set pawoo sources to html page after upload. 2017-05-03 00:09:51 -05:00
evazion
3eafc5ae6a Don't clobber image board sources. 2016-11-19 02:29:33 -06:00
r888888888
f85d5a5570 addresses #2498: revised implementation 2015-08-18 12:45:10 -07:00
r888888888
407523b04c addresses #2498: Protect source downloader against server-side request forgery attacks 2015-08-18 12:26:26 -07:00
Toks
462e1ed324 Implement #2395 for nijie twitter and tumblr 2015-06-13 11:26:47 -04:00
Toks
bb4267af7d remove debug 2015-05-13 18:32:02 -04:00
r888888888
ac6d1fa498 bug fix with pixiv api 2015-05-11 15:53:45 -07:00
r888888888
f12fb40e3e implements #2350 2015-02-20 20:04:23 -08:00
Toks
4231fac4cb #2328: fix image board src rewrite 2014-12-18 11:14:25 -05:00
r888888888
43ee2875bf potential fix for #2328 2014-12-17 14:18:25 -08:00
r888888888
97a71fe5a0 fixes #2298 2014-12-03 12:46:05 -08:00
r888888888
981e6dab67 no longer require presence of ugoira tag to process ugoira posts 2014-10-23 16:34:31 -07:00
evazion
c0a803e3a6 Allow ugoira to be uploaded directly from the zip URL. 
This refactors things such that

* Fetching the ugoira frame data happens at the end of rewrite, not
  rewrite_html_pages. This way uploading zip URLs should work.
* The source data is cached in an instance variable in case it
  was indeed fetched during rewrite_html_pages. This way it doesn't get
  fetched a second time.
 2014-10-19 02:30:03 -07:00
r888888888
3bb06c2be4 integrate ugoiras into zip+webm+preview 2014-10-19 02:30:02 -07:00
r888888888
0a61aac231 refactor 2014-10-19 02:30:02 -07:00