Commit Graph

33 Commits

Author SHA1 Message Date
evazion
28971fe103 sources: factor out site_name method. 2022-03-11 23:20:53 -06:00
evazion
b4aea72d04 sources: remove preview_urls method from base strategy.
Remove the `preview_urls` method from strategies. The only place this was used was
when doing IQDB searches, to download the thumbnail image from the source instead of
the full image.

This wasn't worth it for a few reasons:

* Thumbnails on other sites are sometimes not the size we want, which could affect
  IQDB results.
* Grabbing thumbnails is complex for some sites. You can't always just rewrite the
  image URL. Sometimes it requires extra API calls, which can be slower than just
  grabbing the full image.
* For videos and animations, thumbnails from other sites don't always match our
  thumbnails. We do smart thumbnail generation to try to avoid blank thumbnails, which
  means we don't always pick the first frame, which could affect IQDB results.

API changes:

* /iqdb_queries?search[file_url] now downloads the URL as is without any modification.
  Before it tried to change thumbnail and sample size image URLs to the full version.

* /iqdb_queries?search[url] now returns an error if the URL is for a HTML page that
  contains multiple images. Before it would grab only the first image and silently
  ignore the rest.
2022-03-11 03:22:23 -06:00
nonamethanks
b9c7e467e5 sources: factor out Source::URL::Tumblr
Also adds support for fetching source data from direct image urls when
possible.
2022-03-08 15:06:06 +01:00
evazion
a7dc05ce63 Enable frozen string literals.
Make all string literals immutable by default.
2021-12-14 21:33:27 -06:00
evazion
38c9559fe8 nokogiri: switch to the nokogumbo-based html5 parser.
https://github.com/sparklemotion/nokogiri/blob/main/CHANGELOG.md#1120--2021-08-02
2021-08-30 21:21:27 -05:00
evazion
4b18361aaf tumblr: fix uses of $ in regexes. 2020-08-17 15:31:18 -05:00
evazion
160b099445 sources: use http client from base strategy. 2020-08-17 15:30:01 -05:00
nonamethanks
5615719218 Tumblr: support highest res for new image urls 2020-08-14 01:23:27 +02:00
evazion
7e471fe223 sources: replace HTTParty with Danbooru::Http in http_exists?. 2020-06-21 15:11:56 -05:00
evazion
67a52dbc2d tumblr: support new va.media.tumblr.com urls. 2020-06-19 13:53:35 -05:00
evazion
1aa0f65187 sources: fix rubocop warnings. 2020-06-16 00:10:37 -05:00
nonamethanks
307df3b3e4 Refactor source normalization
* Move the source normalization logic out of the post model
  and into individual sources' strategies.
* Rewrite normalization tests to be handled into each source's test,
  and expand them significantly. Previously we were only testing
  a very small subset of domains and variants.
* Fix up normalization for several sites.
* Normalize fav.me urls into normal deviantart urls.
2020-05-21 22:46:51 +02:00
lllusion3469
65fb0cf510 tumblr: pick biggest image based on resolution
photo[:alt_sizes] may contain a bigger image than
photo[:original_size]

fixes #4324
2020-04-08 16:42:17 +02:00
evazion
309821bf73 rubocop: fix various style issues. 2019-12-22 21:23:37 -06:00
evazion
84ba1d417f Fix #4220: Uploading from Tumblr is broken. 2019-12-15 19:04:52 -06:00
evazion
41378bc8e3 sources: replace HttpartyCache with Danbooru::Http. 2019-12-15 17:06:58 -06:00
evazion
c700ea4b5f Fix #4016: Translated tags failing to find some tags.
* Normalize spaces to underscores when saving other names. Preserve case
  since case can be significant.

* Fix WikiPage#other_names_include to search case-insensitively (note:
  this prevents using the index).

* Fix sources to return the raw tags in `#tags` and the normalized tags
  in `#normalized_tags`. The normalized tags are the tags that will be
  matched against other names.
2018-12-16 11:37:57 -06:00
evazion
5cf6a43918 sources: fix sources sometimes choosing wrong strategy (fix #3968)
Fix sources choosing the wrong strategy when the referer belongs to a
different site (for example, when uploading a twitter post with a pixiv
referer).

* Fix `match?` to only consider the main url, not the referer.

* Change `match?` to match against a list of domains given by the `domains` method.

* Change `match?` to an instance method.
2018-11-04 13:00:17 -06:00
evazion
16b1b72da5 tumblr: fix video urls not being recognized. 2018-10-09 12:44:59 -05:00
evazion
184a5ebf3e tumblr: fix _640 images not being recognized (#3944).
Fixes _640 images not being matched by the IMAGE regex and therefore not
being rewritten to the largest size.
2018-10-09 12:44:59 -05:00
evazion
d874c68419 tumblr: fix image_urls when api data is unavailable. 2018-10-09 12:44:59 -05:00
evazion
b0d7d90103 tumblr: extract info from url when api data is unavailable.
Derive the artist name / profile url / page url from the source URLs when
the API response is unavailable because the Tumblr post was deleted.

This fixes the artist finder to work on bad_tumblr_id posts.
2018-10-09 12:44:59 -05:00
evazion
0c31a5d6a9 tumblr: don't fail when api data is unavailable (#3948).
The api data is unavailable when the work is deleted (bad_tumblr_id), or
when the source is a direct image url with no page referer.
2018-10-09 12:44:59 -05:00
evazion
4c55c809b0 tumblr: don't fail when api key isn't configured. 2018-10-09 12:44:59 -05:00
evazion
8aed1e19b9 tumblr: remove obsolete cdn rewrite rule.
This cdn is no longer used by tumblr.
2018-10-09 12:44:59 -05:00
Albert Yi
4972c998f8 rely on preview urls if available for gallery 2018-09-11 15:06:12 -07:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
evazion
b185efbb5f tumblr commentaries: include asker's name in ask posts (#3586). 2018-03-30 21:42:51 -05:00
evazion
3fefb73e90 Fix #3561: Tumblr: support answer posts. 2018-02-24 10:31:59 -06:00
evazion
f896c24d37 tumblr: fail more gracefully if API key is not configured. 2017-11-18 13:53:11 -06:00
evazion
71f84b10af tumblr: convert commentary to dtext.
* Convert Tumblr commentary to DText.
* Strip extraneous whitespace in links and blockquotes.
* Add newlines after block elements to ensure they're separated from
  subsequent blocks.
2017-07-01 11:15:48 -05:00
evazion
118843b26a tumblr: fetch tags, commentary, and image urls. 2017-06-25 15:34:15 -05:00
Toks
462e1ed324 Implement #2395 for nijie twitter and tumblr 2015-06-13 11:26:47 -04:00