jaculabilis/danbooru
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
7,689 Commits 1 Branch 0 Tags
eade33fa7ce4fb2fca6aaa53d831cebd98225039
Commit Graph
167 Commits
Author SHA1 Message Date
evazion 5cf6a43918 sources: fix sources sometimes choosing wrong strategy (fix #3968) 
Fix sources choosing the wrong strategy when the referer belongs to a
different site (for example, when uploading a twitter post with a pixiv
referer).

* Fix `match?` to only consider the main url, not the referer.

* Change `match?` to match against a list of domains given by the `domains` method.

* Change `match?` to an instance method.
 2018-11-04 13:00:17 -06:00
evazion 53a51310a3 tumblr: add canonical url tests (#3385). 2018-10-09 12:55:48 -05:00
evazion 16b1b72da5 tumblr: fix video urls not being recognized. 2018-10-09 12:44:59 -05:00
evazion 184a5ebf3e tumblr: fix _640 images not being recognized (#3944). 
Fixes _640 images not being matched by the IMAGE regex and therefore not
being rewritten to the largest size.
 2018-10-09 12:44:59 -05:00
evazion d874c68419 tumblr: fix image_urls when api data is unavailable. 2018-10-09 12:44:59 -05:00
evazion b0d7d90103 tumblr: extract info from url when api data is unavailable. 
Derive the artist name / profile url / page url from the source URLs when
the API response is unavailable because the Tumblr post was deleted.

This fixes the artist finder to work on bad_tumblr_id posts.
 2018-10-09 12:44:59 -05:00
evazion 0c31a5d6a9 tumblr: don't fail when api data is unavailable (#3948). 
The api data is unavailable when the work is deleted (bad_tumblr_id), or
when the source is a direct image url with no page referer.
 2018-10-09 12:44:59 -05:00
evazion 4c55c809b0 tumblr: don't fail when api key isn't configured. 2018-10-09 12:44:59 -05:00
evazion c8d538f618 moebooru: delegate to substrategy based on post source (#3911). 
If the yande.re or konachan.com post has a source from a supported site,
for example Pixiv or Twitter, then delegate the artist and commentary
lookup to that substrategy.

Only do this for sources from recognized sites, not the null strategy.
 2018-10-06 14:27:49 -05:00
evazion e5a4193dd4 moebooru: support batch bookmarklet previews (#3911). 2018-10-06 00:58:22 -05:00
evazion fdb6e4ecee moebooru: rewrite konachan urls for Post#normalized_source (#3911). 2018-10-06 00:58:22 -05:00
evazion 864349dc7b moebooru: fetch tags (#3911). 2018-10-06 00:58:22 -05:00
evazion bd3fb7d70e Post#normalized_source: fix for yande.re urls. 
Fix regex for yande.re urls like this:

    https://files.yande.re/image/b66909b940e8d77accab7c9b25aa4dc3/yande.re%20377828.png
 2018-10-01 20:03:21 -05:00
evazion d9063a9f2a nijie: support preview urls (#3919). 2018-09-24 17:08:37 -05:00
evazion 5525bbe1ca nijie: normalize all thumbnail urls (#3919). 2018-09-23 20:08:14 -05:00
evazion d294514dc0 nijie: don't crash on invalid urls or deleted works (#3919). 2018-09-23 20:08:14 -05:00
evazion b6228505aa nijie: fix page_url method. 
The id in a bare image url is the member id, not the illust id.
 2018-09-23 20:08:13 -05:00
evazion 39f9e01b13 nicoseiga: fix canonical_url to use the image url. 2018-09-22 11:07:18 -05:00
evazion 958a9f505b moebooru: rewrite sample urls + support bookmarklet on html page. 
* Fixes #2942: Add Moebooru Rewrite for Sample Images.
* Addresses #3911: Improve Moebooru support.
 2018-09-19 23:32:21 -05:00
evazion 96e89cecfb tests: move twitter canonical url test. 2018-09-17 23:27:53 -05:00
evazion 4a99cb098f moebooru: use the image url as the canonical url. 2018-09-16 21:00:11 -05:00
evazion d9ce953752 Fix #3906: Moebooru strategy raises NotImplementedError. 2018-09-16 21:00:11 -05:00
evazion cae78fa8ee moebooru: move tests from unit/downloads to unit/sources. 2018-09-16 21:00:11 -05:00
evazion f135a7c064 twitter: normalize canonical urls. 
Normalize http://mobile.twitter.com to http://twitter.com in canonical urls.
 2018-09-16 15:03:47 -05:00
evazion bd47641601 twitter: don't fail when api key isn't configured. 2018-09-16 15:03:47 -05:00
evazion 325120ee51 twitter: fix parsing of the artist name from the url. 
Fixes URLs like https://twitter.com/intent/user?user_id=123 being
incorrectly normalized to http://twitter.com/intent/ in artist entries.

Also fixes the artist name to be taken from the url when it can't be
obtained from the api (when the tweet is deleted).
 2018-09-16 15:03:23 -05:00
evazion fbd5f6b7f2 pixiv: fix preview_urls for ugoiras (#3891). 2018-09-12 00:43:10 -05:00
evazion 37fc215d75 pixiv: fix preview_urls to use correct url (#3891). 2018-09-11 23:55:46 -05:00
Albert Yi a5df178bcc Merge pull request #3886 from r888888888/source-api-caching 
cache api clients
 2018-09-11 17:34:25 -07:00
Albert Yi 4972c998f8 rely on preview urls if available for gallery 2018-09-11 15:06:12 -07:00
Albert Yi 266c7c0d5b cache api clients 2018-09-11 14:19:17 -07:00
evazion 9a980367f6 twitter: normalize artist commentaries to nfkc (#3719) 
Fixes hashtags not being interpreted when the author uses a fullwidth
number sign (＃, U+FF03).

ref: https://github.com/r888888888/danbooru/issues/3719#issuecomment-419535610
 2018-09-10 21:45:50 -05:00
evazion c9300cc54e sta.sh: add tests + docs. 2018-09-10 19:26:53 -05:00
evazion a67edb8783 deviantart: fix artist finder for artist names with underscores. 
Fix the artist finder for urls like this:

  https://orig00.deviantart.net/4274/f/2010/230/8/a/pkmn_king_and_queen_by_mikoto_chan.jpg

that don't contain a deviantart id but do contain the artist name.
 2018-09-07 12:23:48 -05:00
evazion 610391205f deviantart: fix artist finder for profile urls missing the 'www'. 
Fix the artist finder to work when the profile url in the artist entry
is missing the 'www'. Example:

  https://deviantart.com/noizave
  https://www.deviantart.com/noizave
 2018-09-07 11:36:48 -05:00
evazion e37844303d deviantart: take artist name from url when unavailable from API. 
In some cases we can't get the artist name from the API, either because
we can't do the API call because the url doesn't contain a deviation id,
or because the work is deleted:

* http://fc08.deviantart.net/files/f/2007/120/c/9/cool_like_me_by_47ness.jpg (work: http://fav.me/dwcohb)
* https://pre00.deviantart.net/423b/th/pre/i/2017/281/e/0/mindflayer_girl01_by_nickbeja-dbpxdt8.png (work: http://fav.me/dbpxd58)

Switch to taking the artist name from the url (when present) to deal
with these cases. Fixes the artist finder and the artist url normalizer
to work in this situation.
 2018-09-03 18:27:01 -05:00
evazion 8f87fb90d9 deviantart: handle urls without deviation ids (fix #3864) 
Some older URL formats don't contain the deviation id:

* http://fc08.deviantart.net/files/f/2007/120/c/9/Cool_Like_Me_by_47ness.jpg
* http://pre06.deviantart.net/8497/th/pre/f/2009/173/c/c/cc9686111dcffffffb5fcfaf0cf069fb.jpg

In these cases we can't make the API call. Fix failures due to not being
able to do API calls in this situation.

Also fix canonical_url to use the image_url when it contains the
deviation id, or the page_url when it doesn't.

Finally, fix page_url to use the url from the API instead of the raw url
given by the user, so that it's in a consistent form for canonical_url.
 2018-09-03 18:26:45 -05:00
evazion 316acead16 deviantart: fix error when uploading image belonging to deleted work. 2018-09-02 23:09:40 -05:00
evazion 2d1b1311d6 deviantart: fix sample urls not being rewritten to full size urls. 2018-09-02 23:09:29 -05:00
evazion 807c3dd5f4 deviantart: remove obsolete image sample rewrite rules. 
Remove rewrite rules for these types of sample urls:

* http://th00.deviantart.net/fs71/PRE/f/2014/065/3/b/goruto_by_xyelkiltrox-d797tit.png
* http://th04.deviantart.net/fs70/300W/f/2009/364/4/d/Alphes_Mimic___Rika_by_Juriesute.png

These URLs aren't served to users any more, and just stripping out "PRE"
or "200H" isn't sufficient to get the full size image. In general, an
api call is required to find the full size image url.
 2018-09-02 14:49:58 -05:00
evazion b9ed676bfb deviantart: handle origin-orig.deviantart.net urls. 2018-09-02 13:57:15 -05:00
evazion d693f01dde Fix #3859: Related tag and find artist don't run when fetch data fails. 
Fixes an exception in the artist finder caused by searching for a nil profile_url.
 2018-09-01 11:48:42 -05:00
evazion d6235d6f9e nijie: add canonical url tests. 2018-08-31 23:23:15 -05:00
evazion 736c22c3ce pixiv: fix fetch source data failing on fanbox images. 
ref: https://danbooru.donmai.us/forum_posts/150000
 2018-08-31 00:21:03 -05:00
evazion a1044dbc19 twitter: fix handling of direct image urls without a referer url. 2018-08-29 17:14:57 -05:00
Albert Yi 762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
Albert Yi 135b97d511 additional fixes for deviantart artist search (#3771) 2018-07-27 12:31:26 -07:00
Albert Yi a0205be8b5 fixes #3771 2018-07-06 11:44:07 -07:00
Albert Yi 5ae37597cd fixes #3728 2018-05-25 13:24:49 -07:00
Albert Yi f3364b9892 skip earlier on failed pixiv tests 2018-05-15 16:15:59 -07:00