jaculabilis/danbooru
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
8,092 Commits 1 Branch 0 Tags
be36968b6d8ae4b2c8d2b2dfd544eada20046bea
Commit Graph

53 Commits

Author SHA1 Message Date
evazion
eba6440b8b Fix #4144: Deviantart Eclipse update broke strategy. 2019-08-28 23:40:29 -05:00
evazion
7f482dc35b deviantart: normalize wixmp.com sources to page urls. 
Normalize sources like this:

   https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/ab917938-d8c1-4b58-933d-0d38c390461f/ddcyi98-cbd39da2-f528-4b26-aadb-a16fe91442b2.jpg/v1/fill/w_1280,h_1760,q_100,strp/tifa_by_chubymi_ddcyi98-fullview.jpg?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7ImhlaWdodCI6Ijw9MTc2MCIsInBhdGgiOiJcL2ZcL2FiOTE3OTM4LWQ4YzEtNGI1OC05MzNkLTBkMzhjMzkwNDYxZlwvZGRjeWk5OC1jYmQzOWRhMi1mNTI4LTRiMjYtYWFkYi1hMTZmZTkxNDQyYjIuanBnIiwid2lkdGgiOiI8PTEyODAifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6aW1hZ2Uub3BlcmF0aW9ucyJdfQ.Uochlma4QJmLwL2ZGMmvTr4HMva4m4bCeF3vnyPSw4I

to this:

   https://www.deviantart.com/chubymi/art/Tifa-807825644

on the sidebar of the posts show page.
 2019-08-06 12:04:05 -05:00
Albert Yi
8a16e089d2 don't try to parse deviant art urls that lack a deviation id in the url 2019-04-16 14:00:38 -07:00
Albert Yi
6f4ca75bb1 try to handle deivantart images hosted on images-wixmp better 2019-04-16 13:55:26 -07:00
Albert Yi
eb8ce2e955 Revert "eliminate old algorithm for deviant art sources, fix canonical_url on deviantart" 
This reverts commit 58ca52b43e.
 2019-04-16 13:22:09 -07:00
Albert Yi
58ca52b43e eliminate old algorithm for deviant art sources, fix canonical_url on deviantart 2019-04-12 16:35:41 -07:00
evazion
811bad5a86 /source.json: include raw api responses in output (#3940). 2018-11-30 00:19:00 -06:00
evazion
e8c3d92696 Fix #3977: DeviantArt: Support new wixmp.com urls. 2018-11-30 00:10:59 -06:00
evazion
8da3474a8b Fix #3994: Some deviantart images mistakenly choosing preview image. 2018-11-29 23:01:08 -06:00
evazion
5cf6a43918 sources: fix sources sometimes choosing wrong strategy (fix #3968) 
Fix sources choosing the wrong strategy when the referer belongs to a
different site (for example, when uploading a twitter post with a pixiv
referer).

* Fix `match?` to only consider the main url, not the referer.

* Change `match?` to match against a list of domains given by the `domains` method.

* Change `match?` to an instance method.
 2018-11-04 13:00:17 -06:00
Albert Yi
9cfbda64bf fixes #3967 2018-11-02 14:21:41 -07:00
evazion
bfed323988 deviantart: fix page_url for when api data is unavailable. 
The api data is unavailable when we can't scrape the uuid, either
because the work is deleted or because the work is actually a sta.sh
upload and we weren't given the sta.sh page in the referer url.
 2018-09-10 19:26:53 -05:00
evazion
7d5d098636 Fix #3877: Add sta.sh strategy. 
Co-authored-by: lllusion3469 <31420484+lllusion3469@users.noreply.github.com>
 2018-09-10 19:26:47 -05:00
evazion
a67edb8783 deviantart: fix artist finder for artist names with underscores. 
Fix the artist finder for urls like this:

  https://orig00.deviantart.net/4274/f/2010/230/8/a/pkmn_king_and_queen_by_mikoto_chan.jpg

that don't contain a deviantart id but do contain the artist name.
 2018-09-07 12:23:48 -05:00
evazion
610391205f deviantart: fix artist finder for profile urls missing the 'www'. 
Fix the artist finder to work when the profile url in the artist entry
is missing the 'www'. Example:

  https://deviantart.com/noizave
  https://www.deviantart.com/noizave
 2018-09-07 11:36:48 -05:00
evazion
e37844303d deviantart: take artist name from url when unavailable from API. 
In some cases we can't get the artist name from the API, either because
we can't do the API call because the url doesn't contain a deviation id,
or because the work is deleted:

* http://fc08.deviantart.net/files/f/2007/120/c/9/cool_like_me_by_47ness.jpg (work: http://fav.me/dwcohb)
* https://pre00.deviantart.net/423b/th/pre/i/2017/281/e/0/mindflayer_girl01_by_nickbeja-dbpxdt8.png (work: http://fav.me/dbpxd58)

Switch to taking the artist name from the url (when present) to deal
with these cases. Fixes the artist finder and the artist url normalizer
to work in this situation.
 2018-09-03 18:27:01 -05:00
evazion
8f87fb90d9 deviantart: handle urls without deviation ids (fix #3864) 
Some older URL formats don't contain the deviation id:

* http://fc08.deviantart.net/files/f/2007/120/c/9/Cool_Like_Me_by_47ness.jpg
* http://pre06.deviantart.net/8497/th/pre/f/2009/173/c/c/cc9686111dcffffffb5fcfaf0cf069fb.jpg

In these cases we can't make the API call. Fix failures due to not being
able to do API calls in this situation.

Also fix canonical_url to use the image_url when it contains the
deviation id, or the page_url when it doesn't.

Finally, fix page_url to use the url from the API instead of the raw url
given by the user, so that it's in a consistent form for canonical_url.
 2018-09-03 18:26:45 -05:00
evazion
316acead16 deviantart: fix error when uploading image belonging to deleted work. 2018-09-02 23:09:40 -05:00
evazion
2d1b1311d6 deviantart: fix sample urls not being rewritten to full size urls. 2018-09-02 23:09:29 -05:00
evazion
807c3dd5f4 deviantart: remove obsolete image sample rewrite rules. 
Remove rewrite rules for these types of sample urls:

* http://th00.deviantart.net/fs71/PRE/f/2014/065/3/b/goruto_by_xyelkiltrox-d797tit.png
* http://th04.deviantart.net/fs70/300W/f/2009/364/4/d/Alphes_Mimic___Rika_by_Juriesute.png

These URLs aren't served to users any more, and just stripping out "PRE"
or "200H" isn't sufficient to get the full size image. In general, an
api call is required to find the full size image url.
 2018-09-02 14:49:58 -05:00
evazion
b9ed676bfb deviantart: handle origin-orig.deviantart.net urls. 2018-09-02 13:57:15 -05:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
Albert Yi
135b97d511 additional fixes for deviantart artist search (#3771) 2018-07-27 12:31:26 -07:00
Albert Yi
4762de65e1 more robust handling of deviant art urls 2018-07-10 14:57:38 -07:00
Albert Yi
a0205be8b5 fixes #3771 2018-07-06 11:44:07 -07:00
evazion
7d295a11d0 fixup! Fix #3654: Deviantart commentaries fail fetching. 2018-04-18 19:01:33 -05:00
evazion
c4ae2bd2fa Fix #3654: Deviantart commentaries fail fetching. 2018-04-18 18:44:56 -05:00
Albert Yi
37ba6714d1 fixes #3617 2018-04-12 12:11:27 -07:00
evazion
ab25e73cd7 deviantart: replace html scraper with api client (#3260). 2018-04-09 20:53:14 -05:00
evazion
04ecca6a06 Fix #3591: DeviantArt commentaries: strip embedded image/text thumbs. 2018-03-31 11:27:14 -05:00
evazion
80219a0ccf commentaries: replace <img> tags with dtext links (#3586) 
Replace <img> tags with dtext links, unless the <img> tag is already
inside an <a> tag.

Also strip Unicode spaces from DeviantArt commentaries.
 2018-03-30 21:42:51 -05:00
evazion
22c9cfcec5 Fix #3384: Fetch commentary not creating fully formed textile links 2017-11-18 13:02:50 -06:00
r888888888
d4008a5ddc fixes #3327 2017-10-09 16:32:05 -07:00
lllusion3469
4c3733ce5a Keep auth_secure cookie from deviantart 
Unlike this projects, all other DA scrapers I found[0] require that both
"auth" and "auth_secure" are set.
While the current approach, for some reason, seems to have worked fine in
the past, it has stopped functioning on some systems, sometimes
permanently (apparently sometimes only temporarily).
I don't have the means to find out the reason behind that, since all
configurations I've tried (mobile, VPN, and regular connection +
4 seperate accounts) resulted in me being effectively logged out and
unable to view mature works after deleting the "auth_secure" cookie.
The "if auth_cookie" part might not actually be necessary, its only
purpose is to prevent unintended breakage due to lack of information
on my part.

[0]: b662b46a44 (diff-aab4e263d591d1c7f23d015a2e540a32R161)
     cbaecf5507 (diff-9f9821e7d744874a2fe42e0a69e4b14aR33)
 2017-08-30 20:47:59 +02:00
r888888888
bc69badadd better error handling of failed deviant art logins 2017-08-28 16:19:41 -07:00
evazion
93d44c6b84 Fix #3224: 403 Forbidden when logging in to DeviantArt. 
Login fails if the User-Agent isn't set while POST'ing the login
credentials.
 2017-07-19 20:25:27 -05:00
evazion
b880b07387 sources: factor out html-to-dtext code to DText.from_html. 2017-07-01 11:15:48 -05:00
evazion
f3f55daea6 Fix #3138: Support DeviantArt login-only works. 2017-06-10 10:34:51 -05:00
evazion
82c7537270 deviantart: force commentary encoding to utf-8. 2017-06-09 13:19:24 -05:00
evazion
ccacfb40be deviantart: translate artist commentary from html to dtext. 2017-06-09 13:19:24 -05:00
Toks
6f17a664e9 Fix deviantart source strategy not working for mature works 2015-09-06 23:36:34 -04:00
Toks
12be880fa9 Support fetching artist commentary from deviantart 2015-09-03 20:03:56 -04:00
Toks
3c92a58585 Deviantart: Fix full-size image not always being uploaded 2015-07-08 00:58:06 -04:00
r888888888
fd74f860ee potential fix for #2404 2015-06-10 17:28:51 -07:00
Toks
854d587373 Fix upload source fetcher fetching from wrong work page for all sites 
e.g. If you were on an html work page on pixiv, clicked a link to a
different html work page on pixiv, and then clicked the bookmarklet,
then it used to fetch the source from the FIRST work you were on instead
of the second.
 2015-06-03 20:59:24 -04:00
Toks
804cbae85c Support fetching deviantart tags 2015-05-25 22:44:55 -04:00
Toks
4b36acdd19 Improve getting deviatnart url from page 
fix #2366
 2015-04-06 18:32:12 -04:00
Toks
04e6aa7474 #1866 update for new deviantart urls 2015-03-13 14:12:17 -04:00
Toks
a3d120c632 #1866: Support HTTPS urls 2014-05-15 23:35:57 -04:00
Toks
0a75402cc7 Support referrer matching for seiga and da 2014-05-08 20:25:11 -04:00