jaculabilis/danbooru
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
9,593 Commits 1 Branch 0 Tags
d3bb5c67ee42cebc80ecfdef4101de15ccbe9708
Commit Graph

75 Commits

Author SHA1 Message Date
evazion
1aa0f65187 sources: fix rubocop warnings. 2020-06-16 00:10:37 -05:00
evazion
d002701bc1 Merge pull request #4494 from nonamethanks/fix_deviantart_api_downloads 
Deviantart: fix api downloads
 2020-06-09 01:37:03 -05:00
nonamethanks
25b801619f Deviantart: fix api downloads 2020-05-31 07:01:43 +02:00
evazion
88d9fc4e5e sources: simplify artist finder url normalization. 
Get rid of `normalized_for_artist_finder?` and `normalizable_for_artist_finder?`.
This was legacy bullshit that was originally designed to avoid API calls
when saving artist entries containing old Pixiv direct image urls that
had already been normalized, or that couldn't be normalized because they
were bad id.

Nowadays we store profile urls in artist entries instead of direct image
urls, so we don't normally need to do any API calls to normalize the
profile url. Strategies should take care to avoid triggering API calls
inside `profile_url` when possible.
 2020-05-29 15:35:15 -05:00
nonamethanks
307df3b3e4 Refactor source normalization 
* Move the source normalization logic out of the post model
  and into individual sources' strategies.
* Rewrite normalization tests to be handled into each source's test,
  and expand them significantly. Previously we were only testing
  a very small subset of domains and variants.
* Fix up normalization for several sites.
* Normalize fav.me urls into normal deviantart urls.
 2020-05-21 22:46:51 +02:00
lllusion3469
45ae8bfb6f deviantart: support non-downloadable videos 2020-05-11 19:51:04 +02:00
lllusion3469
40fa985e26 deviantart: use #at_css instead of #search 
only one result needed, query is css
 2020-05-11 19:51:04 +02:00
lllusion3469
0c180b521c deviantart: avoid download api call if not downloadable 
because it's included in api_response which is part of /source.json
 2020-05-11 19:51:04 +02:00
lllusion3469
70beb7288d rubocop: fix various issues 2020-05-11 19:51:04 +02:00
lllusion3469
0d5e31868f deviantart: fix non-downloadable flash files 2020-05-11 19:51:04 +02:00
lllusion3469
46e9f2dede deviantart: switch to Danbooru::Http 
httprb doesn't seem to support a base_uri parameter so use URI.join with
a relative path instead
 2020-05-11 16:11:15 +02:00
lllusion3469
2794cd254d deviantart: return nil on failure instead of "" 
was also part of eba6440b8b
 2020-05-11 16:11:15 +02:00
lllusion3469
413227e7de deviantart: remove #api_url 
similar change in eba6440b8b

in case of #page it may get rid of the redirect if artist and title are
found
 2020-05-11 16:11:15 +02:00
lllusion3469
c4a403afca deviantart: remove unreachable else 
api_deviation is either #blank? (if condition) or #present?

was also part of eba6440b8b
 2020-05-11 16:11:14 +02:00
lllusion3469
f4b4e12235 deviantart: use image_url as it's a single image 2020-05-11 16:10:56 +02:00
lllusion3469
769bf87a4a deviantart: don't apply /intermediary/ hack for gifs 
gifs are always stored as original anyways so the /intermediary/ url
doesn't actually exist for gifs

example:
https://www.deviantart.com/heartgear/art/Silent-Night-579982816
 2020-05-11 16:10:33 +02:00
lllusion3469
c2e86385a3 deviantart: don't strip metadata 
was also part of eba6440b8b
 2020-05-11 16:10:33 +02:00
lllusion3469
1a49ef46f9 deviantart: cache refresh token for 11 weeks 
it's valid for 3 months according to this:
https://www.deviantart.com/developers/authentication#refresh

use 11 weeks instead to be safe
 2020-05-11 16:10:33 +02:00
lllusion3469
f58564a71f deviantart: don't rewrite download url 
it's all handled through something like
https://api-da.wixmp.com/_api/download/file?downloadToken=$TOKEN
now so those modifications aren't necessary anymore.
In fact, the one to "strip s3 query params" removes the token, breaking
the download url.
 2020-05-11 16:10:32 +02:00
lllusion3469
9205c32424 deviantart: revert to 7f482dc35b 
that's the latest commit made to deviantart files before switching from
the developer API to the Javascript backend from the new "Eclipse"
frontend.
This is necessary because it's basically impossible to download posts
now with the JS backend without being logged in, i.e. having the cookies
from a logged in user, which can't be used for very long even if
exporting them from a browser. You would have to save the cookies
deviantart sends you back via the "Set-Cookie" header in a database
somewhere in addition to the other added complexity.

also
* (temporarily) replace HttpartyCache with HTTParty as it's long been
  removed
* fix one case of "last argument as keyword parameter"
* change repository url (5d1a1cc87e)
* remove self-explanatory comment
 2020-05-11 16:09:00 +02:00
evazion
309821bf73 rubocop: fix various style issues. 2019-12-22 21:23:37 -06:00
evazion
1c3b96dc4e deviantart: fix non-downloadable posts (#4219). 
Adapted from e51e1aa7dd.
 2019-12-06 22:54:08 -06:00
evazion
eba6440b8b Fix #4144: Deviantart Eclipse update broke strategy. 2019-08-28 23:40:29 -05:00
evazion
7f482dc35b deviantart: normalize wixmp.com sources to page urls. 
Normalize sources like this:

   https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/ab917938-d8c1-4b58-933d-0d38c390461f/ddcyi98-cbd39da2-f528-4b26-aadb-a16fe91442b2.jpg/v1/fill/w_1280,h_1760,q_100,strp/tifa_by_chubymi_ddcyi98-fullview.jpg?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7ImhlaWdodCI6Ijw9MTc2MCIsInBhdGgiOiJcL2ZcL2FiOTE3OTM4LWQ4YzEtNGI1OC05MzNkLTBkMzhjMzkwNDYxZlwvZGRjeWk5OC1jYmQzOWRhMi1mNTI4LTRiMjYtYWFkYi1hMTZmZTkxNDQyYjIuanBnIiwid2lkdGgiOiI8PTEyODAifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6aW1hZ2Uub3BlcmF0aW9ucyJdfQ.Uochlma4QJmLwL2ZGMmvTr4HMva4m4bCeF3vnyPSw4I

to this:

   https://www.deviantart.com/chubymi/art/Tifa-807825644

on the sidebar of the posts show page.
 2019-08-06 12:04:05 -05:00
Albert Yi
8a16e089d2 don't try to parse deviant art urls that lack a deviation id in the url 2019-04-16 14:00:38 -07:00
Albert Yi
6f4ca75bb1 try to handle deivantart images hosted on images-wixmp better 2019-04-16 13:55:26 -07:00
Albert Yi
eb8ce2e955 Revert "eliminate old algorithm for deviant art sources, fix canonical_url on deviantart" 
This reverts commit 58ca52b43e.
 2019-04-16 13:22:09 -07:00
Albert Yi
58ca52b43e eliminate old algorithm for deviant art sources, fix canonical_url on deviantart 2019-04-12 16:35:41 -07:00
evazion
811bad5a86 /source.json: include raw api responses in output (#3940). 2018-11-30 00:19:00 -06:00
evazion
e8c3d92696 Fix #3977: DeviantArt: Support new wixmp.com urls. 2018-11-30 00:10:59 -06:00
evazion
8da3474a8b Fix #3994: Some deviantart images mistakenly choosing preview image. 2018-11-29 23:01:08 -06:00
evazion
5cf6a43918 sources: fix sources sometimes choosing wrong strategy (fix #3968) 
Fix sources choosing the wrong strategy when the referer belongs to a
different site (for example, when uploading a twitter post with a pixiv
referer).

* Fix `match?` to only consider the main url, not the referer.

* Change `match?` to match against a list of domains given by the `domains` method.

* Change `match?` to an instance method.
 2018-11-04 13:00:17 -06:00
Albert Yi
9cfbda64bf fixes #3967 2018-11-02 14:21:41 -07:00
evazion
bfed323988 deviantart: fix page_url for when api data is unavailable. 
The api data is unavailable when we can't scrape the uuid, either
because the work is deleted or because the work is actually a sta.sh
upload and we weren't given the sta.sh page in the referer url.
 2018-09-10 19:26:53 -05:00
evazion
7d5d098636 Fix #3877: Add sta.sh strategy. 
Co-authored-by: lllusion3469 <31420484+lllusion3469@users.noreply.github.com>
 2018-09-10 19:26:47 -05:00
evazion
a67edb8783 deviantart: fix artist finder for artist names with underscores. 
Fix the artist finder for urls like this:

  https://orig00.deviantart.net/4274/f/2010/230/8/a/pkmn_king_and_queen_by_mikoto_chan.jpg

that don't contain a deviantart id but do contain the artist name.
 2018-09-07 12:23:48 -05:00
evazion
610391205f deviantart: fix artist finder for profile urls missing the 'www'. 
Fix the artist finder to work when the profile url in the artist entry
is missing the 'www'. Example:

  https://deviantart.com/noizave
  https://www.deviantart.com/noizave
 2018-09-07 11:36:48 -05:00
evazion
e37844303d deviantart: take artist name from url when unavailable from API. 
In some cases we can't get the artist name from the API, either because
we can't do the API call because the url doesn't contain a deviation id,
or because the work is deleted:

* http://fc08.deviantart.net/files/f/2007/120/c/9/cool_like_me_by_47ness.jpg (work: http://fav.me/dwcohb)
* https://pre00.deviantart.net/423b/th/pre/i/2017/281/e/0/mindflayer_girl01_by_nickbeja-dbpxdt8.png (work: http://fav.me/dbpxd58)

Switch to taking the artist name from the url (when present) to deal
with these cases. Fixes the artist finder and the artist url normalizer
to work in this situation.
 2018-09-03 18:27:01 -05:00
evazion
8f87fb90d9 deviantart: handle urls without deviation ids (fix #3864) 
Some older URL formats don't contain the deviation id:

* http://fc08.deviantart.net/files/f/2007/120/c/9/Cool_Like_Me_by_47ness.jpg
* http://pre06.deviantart.net/8497/th/pre/f/2009/173/c/c/cc9686111dcffffffb5fcfaf0cf069fb.jpg

In these cases we can't make the API call. Fix failures due to not being
able to do API calls in this situation.

Also fix canonical_url to use the image_url when it contains the
deviation id, or the page_url when it doesn't.

Finally, fix page_url to use the url from the API instead of the raw url
given by the user, so that it's in a consistent form for canonical_url.
 2018-09-03 18:26:45 -05:00
evazion
316acead16 deviantart: fix error when uploading image belonging to deleted work. 2018-09-02 23:09:40 -05:00
evazion
2d1b1311d6 deviantart: fix sample urls not being rewritten to full size urls. 2018-09-02 23:09:29 -05:00
evazion
807c3dd5f4 deviantart: remove obsolete image sample rewrite rules. 
Remove rewrite rules for these types of sample urls:

* http://th00.deviantart.net/fs71/PRE/f/2014/065/3/b/goruto_by_xyelkiltrox-d797tit.png
* http://th04.deviantart.net/fs70/300W/f/2009/364/4/d/Alphes_Mimic___Rika_by_Juriesute.png

These URLs aren't served to users any more, and just stripping out "PRE"
or "200H" isn't sufficient to get the full size image. In general, an
api call is required to find the full size image url.
 2018-09-02 14:49:58 -05:00
evazion
b9ed676bfb deviantart: handle origin-orig.deviantart.net urls. 2018-09-02 13:57:15 -05:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
Albert Yi
135b97d511 additional fixes for deviantart artist search (#3771) 2018-07-27 12:31:26 -07:00
Albert Yi
4762de65e1 more robust handling of deviant art urls 2018-07-10 14:57:38 -07:00
Albert Yi
a0205be8b5 fixes #3771 2018-07-06 11:44:07 -07:00
evazion
7d295a11d0 fixup! Fix #3654: Deviantart commentaries fail fetching. 2018-04-18 19:01:33 -05:00
evazion
c4ae2bd2fa Fix #3654: Deviantart commentaries fail fetching. 2018-04-18 18:44:56 -05:00
Albert Yi
37ba6714d1 fixes #3617 2018-04-12 12:11:27 -07:00