Commit Graph

52 Commits

Author SHA1 Message Date
evazion
eba6440b8b Fix #4144: Deviantart Eclipse update broke strategy. 2019-08-28 23:40:29 -05:00
evazion
1f73e60514 sources: add methods for customizing new artist entries.
* Rename `unique_id` to `tag_name`.

* Add `other_names` and `profile_urls` methods that sources can override
  to provide extra names or urls when creating new artist entries.
2018-12-27 15:03:11 -06:00
evazion
2170961f47 artists: improve prefilling of new artist form (#4028)
* When creating an artist by clicking the '?' next to the artist tag in
  the tag list, prefill the new artist form by finding the artist's last
  upload and fetching its source data.

  Previously we filled the urls with the source of the artist's last
  upload, which was wrong because it was usually a direct image URL (#3078).

* Fix the other names field not escaping spaces within names to underscores.

* Fix the other names field being potentially prefilled with duplicate names.
2018-12-27 15:03:11 -06:00
evazion
c700ea4b5f Fix #4016: Translated tags failing to find some tags.
* Normalize spaces to underscores when saving other names. Preserve case
  since case can be significant.

* Fix WikiPage#other_names_include to search case-insensitively (note:
  this prevents using the index).

* Fix sources to return the raw tags in `#tags` and the normalized tags
  in `#normalized_tags`. The normalized tags are the tags that will be
  matched against other names.
2018-12-16 11:37:57 -06:00
evazion
811bad5a86 /source.json: include raw api responses in output (#3940). 2018-11-30 00:19:00 -06:00
evazion
308a5021b4 wiki pages: convert other_names to array (#3987). 2018-11-13 19:18:11 -06:00
evazion
628341f7f0 Fix #3969: Translated tags should ignore artist tags. 2018-11-04 17:07:35 -06:00
evazion
5cf6a43918 sources: fix sources sometimes choosing wrong strategy (fix #3968)
Fix sources choosing the wrong strategy when the referer belongs to a
different site (for example, when uploading a twitter post with a pixiv
referer).

* Fix `match?` to only consider the main url, not the referer.

* Change `match?` to match against a list of domains given by the `domains` method.

* Change `match?` to an instance method.
2018-11-04 13:00:17 -06:00
evazion
b0d7d90103 tumblr: extract info from url when api data is unavailable.
Derive the artist name / profile url / page url from the source URLs when
the API response is unavailable because the Tumblr post was deleted.

This fixes the artist finder to work on bad_tumblr_id posts.
2018-10-09 12:44:59 -05:00
evazion
09a8198979 /artists: add wildcard, regex search to url field (#3900)
Allow searching the URL field by regex or by wildcard.

If the query looks like `/twitter/` do a regex search, otherwise if it
looks like `http://www.twitter.com/*` do a wildcard search, otherwise if
it looks like an url do an artist finder search, lastly if it looks like
`twitter` do a `*twitter*` search.
2018-09-21 21:19:01 -05:00
evazion
d9ce953752 Fix #3906: Moebooru strategy raises NotImplementedError. 2018-09-16 21:00:11 -05:00
evazion
bd47641601 twitter: don't fail when api key isn't configured. 2018-09-16 15:03:47 -05:00
evazion
325120ee51 twitter: fix parsing of the artist name from the url.
Fixes URLs like https://twitter.com/intent/user?user_id=123 being
incorrectly normalized to http://twitter.com/intent/ in artist entries.

Also fixes the artist name to be taken from the url when it can't be
obtained from the api (when the tweet is deleted).
2018-09-16 15:03:23 -05:00
evazion
583f8457f0 artists: clean up artist finding logic.
Rename Artist#find_all_by_url to url_matches and drop previous
url_matches method, along with find_artists and search_for_profile.

Previously find_artists tried to lookup the url, referer url, and profile
url in turn until an artist match was found. This was wasteful, because
the source strategy already knows which url to lookup (usually the profile
url). If that url doesn't find a match, then the artist doesn't exist.
2018-09-11 20:14:46 -05:00
Albert Yi
4972c998f8 rely on preview urls if available for gallery 2018-09-11 15:06:12 -07:00
evazion
b924c2bb9c nijie: fix artist url normalization. 2018-09-09 13:17:52 -05:00
evazion
950fcdb7b2 uploads: add new source:<url> dupe check (fix #3873)
* On the /uploads/new page, instead of just showing a "This post has
probably already been uploaded" message, show the actual thumbnails of
posts having the same source as what the user is trying to upload.

* Move the iqdb results section up top, beside the related posts section.
2018-09-06 20:43:20 -05:00
Albert Yi
e695cdde75 add a default for image_url on Sources::Strategies#canonical_url 2018-09-04 11:35:33 -07:00
evazion
d693f01dde Fix #3859: Related tag and find artist don't run when fetch data fails.
Fixes an exception in the artist finder caused by searching for a nil profile_url.
2018-09-01 11:48:42 -05:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
evazion
302994e5d9 Fix #3639: Favorite count pixiv tags aren't skipped by translated tags. 2018-04-13 22:39:52 -05:00
evazion
265377bdbb Fix #3450: Aliased tags show up under translated tags.
Resolves aliases in translated tags. For example, say we lookup `遠坂凛`
and find `tohsaka_rin` and `toosaka_rin`. We apply aliases so that
`tohsaka_rin` becomes `toosaka_rin`, which is then returned as the only
translated tag.
2017-12-23 12:27:58 -06:00
evazion
cbd68ac91b /source.json: do artist lookup and return matches in artists. 2017-11-14 23:28:25 -06:00
evazion
9b031a9c20 Fix #3266: Normalize fullwidth Unicode characters in translated tags. 2017-08-12 17:13:52 -05:00
evazion
cc8986641b Fix #3208: Fix translated tag suggestions for Pixiv.
* Only suggest the Danbooru tag with the same name if there is no
  matching wiki other name. Example: if we have the Pixiv tag `Fate` and
  the Danbooru tag `fate_(series)` with other name `fate`, suggest that,
  not the Danbooru tag `fate`.

* Don't suggest tags that are empty or whose wiki is deleted.

* Only split tags on "/" if there are no other matches, and only for Pixiv.

* For Pixiv, only include traditional media tags in tag list, not digital media (Photoshop, SAI).

* Add some tests.
2017-07-09 12:10:30 -05:00
evazion
fed23c0baa fetch source data: fix gallery warning for non-pixiv galleries. 2017-06-06 23:03:07 -05:00
evazion
757a7da49b fixup! /source.json: don't encode '&' as '&amp;' in commentaries. 2017-05-10 21:03:29 -05:00
evazion
a2a6a0ce27 /source.json: don't encode '&' as '&amp;' in commentaries. 2017-05-10 18:01:52 -05:00
evazion
8015d62b46 /source.json: strip html tags from artist commentary. 2017-05-10 18:01:51 -05:00
evazion
b2f62482a1 Remove artist commentary handling from download rewrite strategies. 2017-05-10 18:01:51 -05:00
Albert Yi
268b7b99ad fix nil error 2017-01-16 15:41:46 -08:00
evazion
5f4535c408 Fix #2839: "Original" tag shows up twice in Pixiv source data 2017-01-14 21:14:06 +00:00
Albert Yi
b0f2b1e8dd Merge pull request #2782 from evazion/fix-2779
Fix #2779: Erroneous pixiv_id parsed for novel/background/profile images
2016-12-05 11:44:38 -08:00
Albert Yi
bd9939a50b some additional documentation about rewrite strategies 2016-12-05 11:43:18 -08:00
evazion
838d6f6eea Fix #2460: Uploading pixiv user's background image throws error.
Fix exception when uploading these types of URLs:

* http://img12.pixiv.net/profile/rapattu/119950.jpg
* http://i2.pixiv.net/img130/profile/minono_aki/8733472.jpg
* http://i2.pixiv.net/img20/img/a-park/novel/3607898.jpg
* http://i1.pixiv.net/novel-cover-original/img/2016/11/03/20/10/58/7436075_f75af69f3eacd1656d3733c72aa959cf.jpg
* http://i1.pixiv.net/background/img/2016/05/17/12/05/48/2074388_d4ac52034f7ca0af3e083d59fde7e97f.jpg

Skip rewriting sources that don't have illust IDs to avoid this
exception. Also fix `tags` so that it doesn't blow up when fetching
source data for these types of sources.
2016-12-04 02:04:44 -06:00
Toks
0a5f1b53ba #2426 refactor to use predetermined referer 2015-07-05 11:02:01 -04:00
r888888888
866347a9f7 fix errors 2015-06-11 21:38:41 -07:00
r888888888
fd74f860ee potential fix for #2404 2015-06-10 17:28:51 -07:00
Toks
854d587373 Fix upload source fetcher fetching from wrong work page for all sites
e.g. If you were on an html work page on pixiv, clicked a link to a
different html work page on pixiv, and then clicked the bookmarklet,
then it used to fetch the source from the FIRST work you were on instead
of the second.
2015-06-03 20:59:24 -04:00
r888888888
2ddc93723c fixes #2327 2015-06-02 19:20:09 -07:00
r888888888
2d266fc0b6 uploads/batch now supports pixiv 2015-03-15 13:56:16 -07:00
r888888888
f12fb40e3e implements #2350 2015-02-20 20:04:23 -08:00
evazion
57bb51621e Add debugger gem.
Fix random VCR failures in Pixiv tests.

Sometimes tests randomly fail because the PHPSESSID they use in their
HTTP requests to Pixiv is different than the one that was originally
recorded by VCR. This causes VCR to complain that the requests don't
match.

This is caused by the PHPSESSID being globally cached in Memcache.
Depending on the order the tests run in (which is random), one set of
tests can use a PHPSESSID that was recorded for a /different/ set of
tests.

Improve Pixiv URL matching.

* Allow URLs that are missing the http:// part. These are sometimes seen
  in artist entries.
* Ignore URLs from random Pixiv domains such as dic.pixiv.net,
  blog.pixiv.net, etc. These are also sometimes in artist entries.

Improve normalize_for_artist_finder! URL matching.

* Normalize www.pixiv.net/stacc/username URLs.
* Correctly normalize URLs that are missing the illust ID part on the end
  (i.e. http://i2.pixiv.net/img04/img/syounen_no_uta/). These are common
  in artist entries.

Match URLs strictly when normalizing for artist entries.

Only normalize Pixiv URLs that strictly match a known format. Pass any
unrecognized URLs through without attempting to normalize them, just to
be safe.

Normalize URLs when saving artist entries.
2014-12-03 13:16:05 -08:00
evazion
7f3b98969f Refactor normalize_for_artist_finder!
Refactors things such that Sources::Site has a normalize_for_artist_finder!
method that delegates to the strategy for the appropriate site. This way
any site that needs to normalize URLs for the artist finder can do so.
2014-10-05 14:11:31 -05:00
Toks
3230ab8781 Add warning when Pixiv post is a gallery of multiple images 2014-06-13 16:33:38 -04:00
小太
cba839ba76 Kill trailing whitespace in ruby files 2013-03-19 23:10:10 +11:00
albert
43177cbad8 fixes #827 2013-03-10 16:56:04 -04:00
albert
2287bc8d61 fixed pixiv sources 2011-09-30 13:27:22 -04:00
albert
19add9aa4e refactored sources 2011-09-29 12:55:15 -04:00
albert
dc31e74809 added tinami support 2011-09-26 17:30:35 -04:00