Commit Graph

385 Commits

Author SHA1 Message Date
lllusion3469
c4a403afca deviantart: remove unreachable else
api_deviation is either #blank? (if condition) or #present?

was also part of eba6440b8b
2020-05-11 16:11:14 +02:00
lllusion3469
f4b4e12235 deviantart: use image_url as it's a single image 2020-05-11 16:10:56 +02:00
lllusion3469
769bf87a4a deviantart: don't apply /intermediary/ hack for gifs
gifs are always stored as original anyways so the /intermediary/ url
doesn't actually exist for gifs

example:
https://www.deviantart.com/heartgear/art/Silent-Night-579982816
2020-05-11 16:10:33 +02:00
lllusion3469
c2e86385a3 deviantart: don't strip metadata
was also part of eba6440b8b
2020-05-11 16:10:33 +02:00
lllusion3469
1a49ef46f9 deviantart: cache refresh token for 11 weeks
it's valid for 3 months according to this:
https://www.deviantart.com/developers/authentication#refresh

use 11 weeks instead to be safe
2020-05-11 16:10:33 +02:00
lllusion3469
f58564a71f deviantart: don't rewrite download url
it's all handled through something like
https://api-da.wixmp.com/_api/download/file?downloadToken=$TOKEN
now so those modifications aren't necessary anymore.
In fact, the one to "strip s3 query params" removes the token, breaking
the download url.
2020-05-11 16:10:32 +02:00
lllusion3469
9205c32424 deviantart: revert to 7f482dc35b
that's the latest commit made to deviantart files before switching from
the developer API to the Javascript backend from the new "Eclipse"
frontend.
This is necessary because it's basically impossible to download posts
now with the JS backend without being logged in, i.e. having the cookies
from a logged in user, which can't be used for very long even if
exporting them from a browser. You would have to save the cookies
deviantart sends you back via the "Set-Cookie" header in a database
somewhere in addition to the other added complexity.

also
* (temporarily) replace HttpartyCache with HTTParty as it's long been
  removed
* fix one case of "last argument as keyword parameter"
* change repository url (5d1a1cc87e)
* remove self-explanatory comment
2020-05-11 16:09:00 +02:00
evazion
e3187e0bd0 tags: add general?, character?, copyright?, artist?, meta?, empty? helper methods. 2020-05-10 23:56:50 -05:00
evazion
f38c38f26e search: split tag_match into user_tag_match / system_tag_match.
When doing a tag search, we have to be careful about which user we're
running the search as because the results depend on the current user.
Specifically, things like private favorites, private favorite groups,
post votes, saved searches, and flagger names depend on the user's
permissions, and whether non-safe or deleted posts are filtered out
depend on whether the user has safe mode on or the hide deleted posts
setting enabled.

* Refactor internal searches to explicitly state whether they're
  running as the system user (DanbooruBot) or as the current user.
* Explicitly pass in the current user to PostQueryBuilder instead of
  implicitly relying on the CurrentUser global.
* Get rid of CurrentUser.admin_mode? (used to ignore the hide deleted
  post setting) and CurrentUser.without_safe_mode (used to ignore safe
  mode).
* Change the /counts/posts.json endpoint to ignore safe mode and the
  hide deleted posts settings when counting posts.
* Fix searches not correctly overriding the hide deleted posts setting
  when multiple status: metatags were used (e.g. `status:banned status:active`)
* Fix fast_count not respecting the hide deleted posts setting when the
  status:banned metatag was used.
2020-05-07 03:29:44 -05:00
nonamethanks
804c87a0ea Artstation: artists can have underscores in their name 2020-04-17 01:10:23 +02:00
lllusion3469
65fb0cf510 tumblr: pick biggest image based on resolution
photo[:alt_sizes] may contain a bigger image than
photo[:original_size]

fixes #4324
2020-04-08 16:42:17 +02:00
BrokenEagle
a45ae09d72 Account for additional Twitter video image links 2020-03-29 19:27:05 +00:00
BrokenEagle
9e16c01285 Allow video thumbnails as direct Twitter images 2020-03-20 20:59:37 +00:00
evazion
ddffffb413 artists: factor out artist finder to separate module. 2020-03-06 23:23:38 -06:00
evazion
49a3538933 pixiv: add support for techorus urls. 2020-03-04 00:00:39 -06:00
evazion
266e4054b0 Fix #4293: ArtStation: use 4k images.
Also fixes #4290 (Image replacements: undefined method hostname for nil:NilClass)
2020-03-03 23:01:29 -06:00
evazion
1244e02fe2 pixiv: handle new https://i-f.pximg.net urls. 2020-02-18 19:22:57 -06:00
BrokenEagle
0569e8346c Fix profile url for normalization when Pawoo errors 2020-01-29 22:27:10 +00:00
evazion
60bf21ff80 twitter: fix preview_urls when source url is a direct image.
Fix preview_urls returning an empty array when the source url is a
direct image from Twitter.

Also return preview_urls in /source.json.
2020-01-21 16:34:03 -06:00
evazion
e42881fbbf Fix #4262: Exception when using Twitter video image links. 2020-01-15 15:20:33 -06:00
evazion
aff3d3b18f Fix various rubocop issues. 2020-01-11 19:01:40 -06:00
BrokenEagle
114637f594 Use more efficient use of regexes for Twitter hashtag normalization 2020-01-10 15:30:54 +00:00
BrokenEagle
2eef73e8fe Add additional Twitter hashtag
- It was mentioned on issue #4047 but was forgotten to be included
2020-01-10 14:52:49 +00:00
evazion
faeec18efc twitter: add hashtag normalization test.
Add test for #4243. Also fix warning from bootsnap:

    iseq.rb:18: warning: nested repeat operator '+' and '?' was replaced with '*' in regular expression: /(?<!\A)生誕祭(?:\d+)?\z/
2020-01-05 17:38:30 -06:00
BrokenEagle
67860a845c Add normalization for Twitter hashtags 2020-01-05 23:01:50 +00:00
evazion
309821bf73 rubocop: fix various style issues. 2019-12-22 21:23:37 -06:00
evazion
84ba1d417f Fix #4220: Uploading from Tumblr is broken. 2019-12-15 19:04:52 -06:00
evazion
41378bc8e3 sources: replace HttpartyCache with Danbooru::Http. 2019-12-15 17:06:58 -06:00
evazion
da84e3a2f2 twitter: replace twitter gem with our own API client.
The twitter gem had several problems:

* It's been unmaintained for over a year.
* It pulled in a lot of dependencies, many of which were outdated. In
  particular, it locked the `http` gem to version 3.3, preventing us
  from upgrading to 4.2.
* It raised exceptions on normal error conditions, like for deleted
  tweets or suspended users, which we really don't want.
* We had to wrap it to provide caching.

Changes:

* Fixes #4226 (Exception when creating new artists entries for suspended
  Twitter accounts)
* Drop support for scraping images from summary cards. Summary cards
  are the previews you get when you link to a website in a tweet. These
  preview images aren't always the best image.
2019-12-13 17:27:03 -06:00
evazion
1c3b96dc4e deviantart: fix non-downloadable posts (#4219).
Adapted from e51e1aa7dd.
2019-12-06 22:54:08 -06:00
evazion
a15bbe4264 moebooru: fix preview_urls to fallback to image_urls.
Fix the moebooru strategy to fallback to returning the image url if we
can't find the preview url. Fixes iqdb lookups failing in some cases
because the strategy didn't return a valid url for preview_url.
2019-10-26 13:51:58 -05:00
evazion
8209a75e95 nicoseiga: remove referer spoofing.
NicoSeiga doesn't appear to have any hotlink protection, so we don't
need to spoof the referer.
2019-10-07 13:15:48 -05:00
evazion
03d9b3feca pixiv: support new https://www.pixiv.net/artworks/:id urls. 2019-09-24 03:33:21 -05:00
evazion
1b426fb23f Fix #4150: Nijie strategy fails for mp4 files. 2019-09-03 22:33:09 -05:00
evazion
64eb6dbb2a pixiv: possible fix for #4152. 2019-09-02 13:13:58 -05:00
evazion
e781c6b608 pixiv: temp disable source strategy (#4152).
Disable use of the Pixiv API until we get it working again.
2019-09-02 11:13:00 -05:00
evazion
eba6440b8b Fix #4144: Deviantart Eclipse update broke strategy. 2019-08-28 23:40:29 -05:00
evazion
7f482dc35b deviantart: normalize wixmp.com sources to page urls.
Normalize sources like this:

   https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/ab917938-d8c1-4b58-933d-0d38c390461f/ddcyi98-cbd39da2-f528-4b26-aadb-a16fe91442b2.jpg/v1/fill/w_1280,h_1760,q_100,strp/tifa_by_chubymi_ddcyi98-fullview.jpg?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7ImhlaWdodCI6Ijw9MTc2MCIsInBhdGgiOiJcL2ZcL2FiOTE3OTM4LWQ4YzEtNGI1OC05MzNkLTBkMzhjMzkwNDYxZlwvZGRjeWk5OC1jYmQzOWRhMi1mNTI4LTRiMjYtYWFkYi1hMTZmZTkxNDQyYjIuanBnIiwid2lkdGgiOiI8PTEyODAifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6aW1hZ2Uub3BlcmF0aW9ucyJdfQ.Uochlma4QJmLwL2ZGMmvTr4HMva4m4bCeF3vnyPSw4I

to this:

   https://www.deviantart.com/chubymi/art/Tifa-807825644

on the sidebar of the posts show page.
2019-08-06 12:04:05 -05:00
evazion
a8896b664d twitter: fix batch bookmarklet selecting wrong image.
Fix regression in 7e465aeda.

https://danbooru.donmai.us/forum_topics/9127?page=276#forum_post_158779
2019-08-06 10:42:45 -05:00
evazion
7e465aedae Fix #4110: New Twitter image urls are broken in bookmarklet. 2019-08-04 20:23:10 -05:00
evazion
9ecf36585c nijie: update for new image urls.
Nijie moved from this:

    https://pic03.nijie.info/nijie_picture/236014_20170620101426_0.png (page: https://www.nijie.info/view.php?id=218856)

to this:

    https://pic.nijie.net/03/nijie_picture/236014_20170620101426_0.png (page: https://www.nijie.info/view.php?id=218856)
2019-08-04 17:49:54 -05:00
Albert Yi
8a16e089d2 don't try to parse deviant art urls that lack a deviation id in the url 2019-04-16 14:00:38 -07:00
Albert Yi
6f4ca75bb1 try to handle deivantart images hosted on images-wixmp better 2019-04-16 13:55:26 -07:00
Albert Yi
eb8ce2e955 Revert "eliminate old algorithm for deviant art sources, fix canonical_url on deviantart"
This reverts commit 58ca52b43e.
2019-04-16 13:22:09 -07:00
Albert Yi
58ca52b43e eliminate old algorithm for deviant art sources, fix canonical_url on deviantart 2019-04-12 16:35:41 -07:00
Albert Yi
d8d4a5ae6f refactor nico seiga manga support 2019-02-25 15:53:07 -08:00
Albert Yi
90ce42a537 add support for nico seiga manga (fixes #4060) 2019-02-25 14:44:45 -08:00
evazion
13dff046f7 pixiv: fix illust id parsing (fixup 8cadef2dd) 2019-01-13 15:02:51 -06:00
evazion
8cadef2dd7 pixiv: fix illust id parsing (fix #4043).
* Tighten up illust id parsing to avoid misparsing ids from
  non-illust urls (sketch urls and novel urls).

* Move id parsing tests from post_test.rb to sources/pixiv_test.rb.

* Drop support for touch.pixiv.net urls. These urls are no longer used
  by Pixiv and aren't present as the source of any posts on Danbooru.
2019-01-13 14:28:51 -06:00
evazion
04d5b16da7 pixiv: fix failure to upload bad pixiv id images (fix #4031)
Bug: Uploading bad pixiv id images failed because the pixiv strategy
raised a BadIDError exception when the upload service checked for the
ugoira frame data.
2019-01-03 18:01:20 -06:00