jaculabilis/danbooru
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
11,646 Commits 1 Branch 0 Tags
f7794de0b7791f0008d239675e4aaf90fabe2e9a
Commit Graph

2840 Commits

Author SHA1 Message Date
evazion
6fe883c316 posts index: clean up limit:<n> / order:random parsing (#2894) 
* Move the limit:<n> / order:random metatag parsing from the controller
  to the post set.

* Introduce `Tag.has_metatag?` and use it to parse these metatags
  instead of using a regex (#2894).
 2018-09-20 19:23:47 -05:00
evazion
958a9f505b moebooru: rewrite sample urls + support bookmarklet on html page. 
* Fixes #2942: Add Moebooru Rewrite for Sample Images.
* Addresses #3911: Improve Moebooru support.
 2018-09-19 23:32:21 -05:00
evazion
f4e08ef30d Downloads::File: fix SSRF inside is_cloudflare? (#2498). 
Fixes the banned IP check not being applied when sending the HEAD
request for is_cloudflare?.

Also fixes the `#size` method not using the uncached url (which meant
the bookmarklet could report the wrong filesize on artstation uploads).
 2018-09-19 20:11:53 -05:00
Albert Yi
9e1e73ec4c migrate blacklists during mass updates (#2251) 2018-09-19 16:25:06 -07:00
evazion
2f17082e73 Downloads::File: fix SSRF when following redirects (#2498). 
Fixes the banned IP check not being applied when following redirects:

  http://danbooru.donmai.us/uploads/new?url=http://httpbin.org/redirect-to%3Furl=http://127.0.0.1/test.jpg
 2018-09-18 16:00:03 -05:00
evazion
99221e4028 Downloads::File: fix SSRF attack when fetching remote size (#2498). 
Fixes the banned IP check not being applied when fetching the remote
file size. This allowed one to trick Danbooru into sending HEAD requests
to private IPs:

  http://danbooru.donmai.us/uploads/new?url=http://127.0.0.1/test.jpg
 2018-09-18 12:16:27 -05:00
evazion
488b1fd994 Downloads::File: memoize strategy. 
Also remove unused options param from constructor
 2018-09-18 10:22:15 -05:00
evazion
9cdfbba6c2 Fix #3910: Corrupted images during upload. 
Use a fresh tempfile for each download attempt instead of reusing the same
file (and having to rewind/truncate it after each failed attempt).
 2018-09-18 10:01:44 -05:00
evazion
d3c135ec72 Downloads::File#http_get_streaming: clean up retry logic. 
Replace handrolled retry logic with retriable gem (already pulled in by another gem).
 2018-09-18 09:44:15 -05:00
evazion
2d5f6b8a35 Fix #3902: Add source of tag autocomplete to the results. 2018-09-17 18:07:48 -05:00
evazion
4a99cb098f moebooru: use the image url as the canonical url. 2018-09-16 21:00:11 -05:00
evazion
d9ce953752 Fix #3906: Moebooru strategy raises NotImplementedError. 2018-09-16 21:00:11 -05:00
evazion
f135a7c064 twitter: normalize canonical urls. 
Normalize http://mobile.twitter.com to http://twitter.com in canonical urls.
 2018-09-16 15:03:47 -05:00
evazion
bd47641601 twitter: don't fail when api key isn't configured. 2018-09-16 15:03:47 -05:00
evazion
325120ee51 twitter: fix parsing of the artist name from the url. 
Fixes URLs like https://twitter.com/intent/user?user_id=123 being
incorrectly normalized to http://twitter.com/intent/ in artist entries.

Also fixes the artist name to be taken from the url when it can't be
obtained from the api (when the tweet is deleted).
 2018-09-16 15:03:23 -05:00
Albert Yi
6dd64da9ba fix invalid content type leaving upload stuck in preprocessing status (#3896) 2018-09-13 13:50:18 -07:00
Albert Yi
7c6ef9f22c Revert "Revert "add prefix index on tags (#3858)"" 
This reverts commit 6210c1554c.
 2018-09-13 11:44:07 -07:00
Albert Yi
6210c1554c Revert "add prefix index on tags (#3858)" 
This reverts commit 8c564f055f.
 2018-09-12 16:41:52 -07:00
Albert Yi
10e9daccfd enable autocorrect search for everyone 2018-09-12 14:14:31 -07:00
Albert Yi
8c564f055f add prefix index on tags (#3858) 2018-09-12 14:01:21 -07:00
Albert Yi
f487b2a2c6 Merge pull request #3889 from evazion/fix-replace-artist-finder 
Cleanup artist finder
 2018-09-12 11:44:42 -07:00
evazion
fbd5f6b7f2 pixiv: fix preview_urls for ugoiras (#3891). 2018-09-12 00:43:10 -05:00
evazion
37fc215d75 pixiv: fix preview_urls to use correct url (#3891). 2018-09-11 23:55:46 -05:00
evazion
583f8457f0 artists: clean up artist finding logic. 
Rename Artist#find_all_by_url to url_matches and drop previous
url_matches method, along with find_artists and search_for_profile.

Previously find_artists tried to lookup the url, referer url, and profile
url in turn until an artist match was found. This was wasteful, because
the source strategy already knows which url to lookup (usually the profile
url). If that url doesn't find a match, then the artist doesn't exist.
 2018-09-11 20:14:46 -05:00
Albert Yi
a5df178bcc Merge pull request #3886 from r888888888/source-api-caching 
cache api clients
 2018-09-11 17:34:25 -07:00
Albert Yi
4972c998f8 rely on preview urls if available for gallery 2018-09-11 15:06:12 -07:00
Albert Yi
20f4dd6d2e fix cache key (#3886) 2018-09-11 14:19:17 -07:00
Albert Yi
266c7c0d5b cache api clients 2018-09-11 14:19:17 -07:00
Albert Yi
f16c3a3f40 fix nijie specs 2018-09-11 13:27:00 -07:00
Albert Yi
e58172483b update iqdb after post replacements (#3885) 2018-09-11 12:24:15 -07:00
Albert Yi
42f7b17a30 Merge pull request #3883 from evazion/refactor-anonymous-user 
Refactor AnonymousUser
 2018-09-11 12:09:25 -07:00
evazion
9a980367f6 twitter: normalize artist commentaries to nfkc (#3719) 
Fixes hashtags not being interpreted when the author uses a fullwidth
number sign (＃, U+FF03).

ref: https://github.com/r888888888/danbooru/issues/3719#issuecomment-419535610
 2018-09-10 21:45:50 -05:00
evazion
bfed323988 deviantart: fix page_url for when api data is unavailable. 
The api data is unavailable when we can't scrape the uuid, either
because the work is deleted or because the work is actually a sta.sh
upload and we weren't given the sta.sh page in the referer url.
 2018-09-10 19:26:53 -05:00
evazion
c9300cc54e sta.sh: add tests + docs. 2018-09-10 19:26:53 -05:00
evazion
7d5d098636 Fix #3877: Add sta.sh strategy. 
Co-authored-by: lllusion3469 <31420484+lllusion3469@users.noreply.github.com>
 2018-09-10 19:26:47 -05:00
evazion
e546e52bd7 Replace AnonymousUser with User.new. 
* Replace AnonymousUser null object with a readonly, unpersisted User object.

* Default always_resize_images to true (previously it was true for
  anonymous users, but false for new members).

* Default comment_threshold to -1 for anonymous users (previously it was
  0 for anonymous but -1 for new members).
 2018-09-09 21:53:08 -05:00
evazion
cb2d85d925 twitter: fix profile_url for twitter.com/i/web/status/:id urls. 
Fix profile_url returning nil for https://twitter.com/i/web/status/943446161586733056.
 2018-09-09 19:48:34 -05:00
evazion
b924c2bb9c nijie: fix artist url normalization. 2018-09-09 13:17:52 -05:00
Albert Yi
b1a9337897 Merge pull request #3875 from evazion/fix-3873 
Fix #3873: Batch bookmarklet for tumblr reports wrong posts as already uploaded
 2018-09-07 14:15:24 -07:00
evazion
a67edb8783 deviantart: fix artist finder for artist names with underscores. 
Fix the artist finder for urls like this:

  https://orig00.deviantart.net/4274/f/2010/230/8/a/pkmn_king_and_queen_by_mikoto_chan.jpg

that don't contain a deviantart id but do contain the artist name.
 2018-09-07 12:23:48 -05:00
evazion
610391205f deviantart: fix artist finder for profile urls missing the 'www'. 
Fix the artist finder to work when the profile url in the artist entry
is missing the 'www'. Example:

  https://deviantart.com/noizave
  https://www.deviantart.com/noizave
 2018-09-07 11:36:48 -05:00
evazion
950fcdb7b2 uploads: add new source:<url> dupe check (fix #3873) 
* On the /uploads/new page, instead of just showing a "This post has
probably already been uploaded" message, show the actual thumbnails of
posts having the same source as what the user is trying to upload.

* Move the iqdb results section up top, beside the related posts section.
 2018-09-06 20:43:20 -05:00
evazion
07c0695ac1 uploads: remove "this post was probably already uploaded" dupe check. 
Also makes it so that using the bookmarklet always triggers async upload
preprocessing. Before it was only triggered when the source passed a
dupe check, but that check was inaccurate (#3873).
 2018-09-06 20:43:04 -05:00
evazion
c2cf6a5441 Fix #3871: Autocomplete can return duplicate tags. 2018-09-05 18:56:30 -05:00
Albert Yi
69b7c9efd0 Merge pull request #3838 from evazion/fix-3837 
Fix #3837: Remove expired bans.
 2018-09-05 12:03:46 -07:00
Albert Yi
a774c6753b show only 10 results in tag autocomplete 2018-09-05 11:17:22 -07:00
Albert Yi
f88cb1340a tweak result counts for autocomplete 2018-09-05 11:08:22 -07:00
Albert Yi
3025db66d7 switch fuzzy autocomplete to spellcorrect intent 2018-09-05 09:41:25 -07:00
Albert Yi
865e3dc976 switch back to similarity function for fuzzy search 2018-09-04 18:23:49 -07:00
evazion
5c457fbe51 pixiv: remove obsolete edgesuite.net rewrite rule. 
This CDN hasn't been seen for several years.

ref: https://danbooru.donmai.us/forum_topics/10766
 2018-09-04 18:15:21 -05:00