Commit Graph

118 Commits

Author SHA1 Message Date
evazion
185693b99b Merge branch 'master' into fix-pixiv-profile-url 2020-06-24 00:06:55 -05:00
evazion
5604ab0079 pixiv: remove fanbox support.
This is broken and it needs to be rewritten as a separate source
strategy anyway.
2020-06-21 11:59:51 -05:00
BrokenEagle
158a4aa916 Fix Pixiv user profile URL to use the latest format
This will only affect new artist and commentary records going forward.
2020-06-17 07:07:33 +00:00
evazion
1aa0f65187 sources: fix rubocop warnings. 2020-06-16 00:10:37 -05:00
evazion
88d9fc4e5e sources: simplify artist finder url normalization.
Get rid of `normalized_for_artist_finder?` and `normalizable_for_artist_finder?`.
This was legacy bullshit that was originally designed to avoid API calls
when saving artist entries containing old Pixiv direct image urls that
had already been normalized, or that couldn't be normalized because they
were bad id.

Nowadays we store profile urls in artist entries instead of direct image
urls, so we don't normally need to do any API calls to normalize the
profile url. Strategies should take care to avoid triggering API calls
inside `profile_url` when possible.
2020-05-29 15:35:15 -05:00
nonamethanks
307df3b3e4 Refactor source normalization
* Move the source normalization logic out of the post model
  and into individual sources' strategies.
* Rewrite normalization tests to be handled into each source's test,
  and expand them significantly. Previously we were only testing
  a very small subset of domains and variants.
* Fix up normalization for several sites.
* Normalize fav.me urls into normal deviantart urls.
2020-05-21 22:46:51 +02:00
evazion
49a3538933 pixiv: add support for techorus urls. 2020-03-04 00:00:39 -06:00
evazion
1244e02fe2 pixiv: handle new https://i-f.pximg.net urls. 2020-02-18 19:22:57 -06:00
evazion
309821bf73 rubocop: fix various style issues. 2019-12-22 21:23:37 -06:00
evazion
03d9b3feca pixiv: support new https://www.pixiv.net/artworks/:id urls. 2019-09-24 03:33:21 -05:00
evazion
64eb6dbb2a pixiv: possible fix for #4152. 2019-09-02 13:13:58 -05:00
evazion
e781c6b608 pixiv: temp disable source strategy (#4152).
Disable use of the Pixiv API until we get it working again.
2019-09-02 11:13:00 -05:00
evazion
13dff046f7 pixiv: fix illust id parsing (fixup 8cadef2dd) 2019-01-13 15:02:51 -06:00
evazion
8cadef2dd7 pixiv: fix illust id parsing (fix #4043).
* Tighten up illust id parsing to avoid misparsing ids from
  non-illust urls (sketch urls and novel urls).

* Move id parsing tests from post_test.rb to sources/pixiv_test.rb.

* Drop support for touch.pixiv.net urls. These urls are no longer used
  by Pixiv and aren't present as the source of any posts on Danbooru.
2019-01-13 14:28:51 -06:00
evazion
04d5b16da7 pixiv: fix failure to upload bad pixiv id images (fix #4031)
Bug: Uploading bad pixiv id images failed because the pixiv strategy
raised a BadIDError exception when the upload service checked for the
ugoira frame data.
2019-01-03 18:01:20 -06:00
evazion
2129e60b2b pixiv: include stacc url in new artist entries (#4028). 2018-12-27 15:03:11 -06:00
evazion
1f73e60514 sources: add methods for customizing new artist entries.
* Rename `unique_id` to `tag_name`.

* Add `other_names` and `profile_urls` methods that sources can override
  to provide extra names or urls when creating new artist entries.
2018-12-27 15:03:11 -06:00
evazion
c700ea4b5f Fix #4016: Translated tags failing to find some tags.
* Normalize spaces to underscores when saving other names. Preserve case
  since case can be significant.

* Fix WikiPage#other_names_include to search case-insensitively (note:
  this prevents using the index).

* Fix sources to return the raw tags in `#tags` and the normalized tags
  in `#normalized_tags`. The normalized tags are the tags that will be
  matched against other names.
2018-12-16 11:37:57 -06:00
evazion
5cf6a43918 sources: fix sources sometimes choosing wrong strategy (fix #3968)
Fix sources choosing the wrong strategy when the referer belongs to a
different site (for example, when uploading a twitter post with a pixiv
referer).

* Fix `match?` to only consider the main url, not the referer.

* Change `match?` to match against a list of domains given by the `domains` method.

* Change `match?` to an instance method.
2018-11-04 13:00:17 -06:00
r888888888
e060236fb7 add exception for direct links to pixiv fanbox images 2018-10-20 23:58:19 -07:00
Albert Yi
a85f3773e3 fix nil commentary case for pixiv strategy #3948 2018-10-12 14:35:44 -07:00
evazion
fbd5f6b7f2 pixiv: fix preview_urls for ugoiras (#3891). 2018-09-12 00:43:10 -05:00
evazion
37fc215d75 pixiv: fix preview_urls to use correct url (#3891). 2018-09-11 23:55:46 -05:00
Albert Yi
4972c998f8 rely on preview urls if available for gallery 2018-09-11 15:06:12 -07:00
evazion
950fcdb7b2 uploads: add new source:<url> dupe check (fix #3873)
* On the /uploads/new page, instead of just showing a "This post has
probably already been uploaded" message, show the actual thumbnails of
posts having the same source as what the user is trying to upload.

* Move the iqdb results section up top, beside the related posts section.
2018-09-06 20:43:20 -05:00
evazion
5c457fbe51 pixiv: remove obsolete edgesuite.net rewrite rule.
This CDN hasn't been seen for several years.

ref: https://danbooru.donmai.us/forum_topics/10766
2018-09-04 18:15:21 -05:00
evazion
4bbe09762d pixiv: remove dead methods (#is_manga?, #page_count, #page). 2018-09-04 18:15:21 -05:00
Albert Yi
8ec96f42f7 fix specs 2018-09-04 13:38:09 -07:00
Albert Yi
4a56f8d160 fixes #3856 for pixiv fanbox urls 2018-09-04 12:53:58 -07:00
evazion
c689a161f6 pixiv: fix failure when normalizing pixiv stacc artist urls (#3856). 2018-08-30 19:24:44 -05:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
Albert Yi
5ae37597cd fixes #3728 2018-05-25 13:24:49 -07:00
Albert Yi
c97b0245d6 reduce expiry for cached pixiv tokens to 1 week, revert to old method for extracting image url from page (fixes #3722) 2018-05-25 10:04:28 -07:00
Albert Yi
bd49f5ed20 rely on pixiv api for getting image url (fixes #3721)
[skip ci]
2018-05-22 09:46:37 -07:00
Albert Yi
be54741dba bubble up source errors in pixiv strategy in test env 2018-05-16 17:19:20 -07:00
evazion
302994e5d9 Fix #3639: Favorite count pixiv tags aren't skipped by translated tags. 2018-04-13 22:39:52 -05:00
evazion
a14c492020 Fix #3396: Pixiv novel cover IDs are parsed incorrectly. 2017-11-21 13:03:03 -06:00
evazion
f8581c8113 Fix #3321: HTML tags not converted in artist commentaries. 2017-11-21 02:42:49 -06:00
r888888888
d124ff8128 fix case when fetching pixiv sources when illust_id param comes before mode 2017-08-23 17:50:33 -07:00
evazion
c270954991 Fix #3258: Fetch source data broken for https://i.pximg.net/* ugoiras. 2017-08-06 17:23:29 -05:00
evazion
80e8c9d174 strategies/pixiv.rb: removed unused methods (#3206).
The page count and the moniker are now obtained from the API.
2017-07-26 18:17:04 -05:00
evazion
cc8986641b Fix #3208: Fix translated tag suggestions for Pixiv.
* Only suggest the Danbooru tag with the same name if there is no
  matching wiki other name. Example: if we have the Pixiv tag `Fate` and
  the Danbooru tag `fate_(series)` with other name `fate`, suggest that,
  not the Danbooru tag `fate`.

* Don't suggest tags that are empty or whose wiki is deleted.

* Only split tags on "/" if there are no other matches, and only for Pixiv.

* For Pixiv, only include traditional media tags in tag list, not digital media (Photoshop, SAI).

* Add some tests.
2017-07-09 12:10:30 -05:00
evazion
94e48aa6f3 pixiv: convert commentary to dtext.
Convert pixiv links to dtext in this manner:

* https://www.pixiv.net/member_illust.php?mode=medium&illust_id=1234
  => pixiv #1234 "»":[/posts?tags=pixiv:1234]

* https://www.pixiv.net/member.php?id=1234
  => "user/1234":[https://www.pixiv.net/member.php?id=1234] "»":[/artists?search[url_matches]=https://www.pixiv.net/member.php?id=1234]
2017-06-16 18:42:37 -05:00
Albert Yi
2f1faf3c38 Merge pull request #3060 from evazion/fix-3048
Fix #3048: Artist Lookup Failing Due to Changing Usernames
2017-05-15 16:33:53 -07:00
Albert Yi
7d2e1f871b Merge pull request #3056 from evazion/fix-3054
Fix #3054: Artist link broken on Post Upload page
2017-05-15 16:17:26 -07:00
evazion
eac827973b Normalize pixiv artist urls to pixiv member page (fix #3048). 2017-05-15 14:12:08 -05:00
evazion
6ee3b2635a Fix returning incorrect pixiv profile url (fixes #3054). 2017-05-14 22:00:06 -05:00
evazion
61a37a09cc Fix translated tags for pixiv (fix #3046).
Translated tags expects `tags` to be an array of `[tag_name, search_link]` pairs.
2017-05-13 19:17:33 -05:00
r888888888
7ae18d1068 bug fix for source pixiv 2017-05-12 16:42:25 -07:00
r888888888
bcec293efe pull additional data from pixiv api (fixes #3042) 2017-05-12 11:17:31 -07:00