Commit Graph

25 Commits

Author SHA1 Message Date
evazion
7e471fe223 sources: replace HTTParty with Danbooru::Http in http_exists?. 2020-06-21 15:11:56 -05:00
evazion
1aa0f65187 sources: fix rubocop warnings. 2020-06-16 00:10:37 -05:00
evazion
88d9fc4e5e sources: simplify artist finder url normalization.
Get rid of `normalized_for_artist_finder?` and `normalizable_for_artist_finder?`.
This was legacy bullshit that was originally designed to avoid API calls
when saving artist entries containing old Pixiv direct image urls that
had already been normalized, or that couldn't be normalized because they
were bad id.

Nowadays we store profile urls in artist entries instead of direct image
urls, so we don't normally need to do any API calls to normalize the
profile url. Strategies should take care to avoid triggering API calls
inside `profile_url` when possible.
2020-05-29 15:35:15 -05:00
evazion
2c60a51f64 Merge pull request #4475 from nonamethanks/refactor_source_normalizing
Refactor source normalization
2020-05-27 16:52:17 -05:00
BrokenEagle
2d88569fac Fixes issues with Artstation source strategy
The reason that the download was failing was not because the 4k size
didn't exist, but because the Artstation had no way to handle image
cover URLs. This caused it to pass nil to the download function.

Additionally, there was no way to get the preview URL size, i.e. the
smallest available image for an Arstation image URL.

- Adds support for cover URLs
- Adds support for preview URL size
2020-05-24 00:38:54 +00:00
nonamethanks
307df3b3e4 Refactor source normalization
* Move the source normalization logic out of the post model
  and into individual sources' strategies.
* Rewrite normalization tests to be handled into each source's test,
  and expand them significantly. Previously we were only testing
  a very small subset of domains and variants.
* Fix up normalization for several sites.
* Normalize fav.me urls into normal deviantart urls.
2020-05-21 22:46:51 +02:00
nonamethanks
804c87a0ea Artstation: artists can have underscores in their name 2020-04-17 01:10:23 +02:00
evazion
266e4054b0 Fix #4293: ArtStation: use 4k images.
Also fixes #4290 (Image replacements: undefined method hostname for nil:NilClass)
2020-03-03 23:01:29 -06:00
evazion
309821bf73 rubocop: fix various style issues. 2019-12-22 21:23:37 -06:00
evazion
41378bc8e3 sources: replace HttpartyCache with Danbooru::Http. 2019-12-15 17:06:58 -06:00
evazion
b12e8b5058 Fix #4030: Artstation artist_name broken for direct image urls. 2018-12-31 11:45:50 -06:00
Albert Yi
ca7c56f95f fix: url normalization for non-project artist subdomains (maybe fixes #4021) 2018-12-19 15:19:00 -08:00
evazion
c700ea4b5f Fix #4016: Translated tags failing to find some tags.
* Normalize spaces to underscores when saving other names. Preserve case
  since case can be significant.

* Fix WikiPage#other_names_include to search case-insensitively (note:
  this prevents using the index).

* Fix sources to return the raw tags in `#tags` and the normalized tags
  in `#normalized_tags`. The normalized tags are the tags that will be
  matched against other names.
2018-12-16 11:37:57 -06:00
evazion
fd2d56a388 Fix #3996: Some direct artstation urls cause the bookmarklet to fail.
* Don't fail on urls that don't contain the project id (direct image urls).
* Don't fail when the work is deleted.
* Parse artist name from url when possible. This way the artist finder works on bad_artstation_id posts.
* Set canonical source url to `https://artist.artstation.com/projects/12345` instead of
  `https://www.artstation.com/artwork/1235` (this way we preserve the artist name).
* Cache api call.
* Include api call results in /source.json.
2018-12-03 15:48:29 -06:00
evazion
5cf6a43918 sources: fix sources sometimes choosing wrong strategy (fix #3968)
Fix sources choosing the wrong strategy when the referer belongs to a
different site (for example, when uploading a twitter post with a pixiv
referer).

* Fix `match?` to only consider the main url, not the referer.

* Change `match?` to match against a list of domains given by the `domains` method.

* Change `match?` to an instance method.
2018-11-04 13:00:17 -06:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
evazion
28cd633210 Fix #3637: Bookmarklet fails on artstation for some posts. 2018-04-13 11:18:09 -05:00
r888888888
3c524aa5e6 fix bug with source::site 2017-11-17 17:26:33 -08:00
r888888888
97ce9cafac add httparty option config 2017-07-12 15:52:48 -07:00
r888888888
eb6c5e3af5 switch to httparty 2017-07-12 15:52:48 -07:00
evazion
b1171306a1 artstation: include only images in image_urls. 2017-06-07 10:20:18 -05:00
evazion
a68ff3c347 artstation: accept www.artstation.com/artwork/foo-bar-baz style urls. 2017-06-07 10:20:17 -05:00
evazion
666aaef69c Add source support for https://$artist.artstation.com/projects/$id. 2017-06-05 14:03:38 -05:00
r888888888
9996030496 include default referer url for art station strategy 2017-02-06 17:37:45 -08:00
r888888888
7a2f652152 add source strategy for artstation 2017-02-06 17:07:09 -08:00