Commit Graph

228 Commits

Author SHA1 Message Date
r888888888
59cd4d71d3 fixes #3467 2017-12-28 14:56:03 -08:00
evazion
48b67967b5 Fix #3457: Timeout when fetching source data from NicoSeiga. 2017-12-27 13:08:14 -06:00
evazion
265377bdbb Fix #3450: Aliased tags show up under translated tags.
Resolves aliases in translated tags. For example, say we lookup `遠坂凛`
and find `tohsaka_rin` and `toosaka_rin`. We apply aliases so that
`tohsaka_rin` becomes `toosaka_rin`, which is then returned as the only
translated tag.
2017-12-23 12:27:58 -06:00
r888888888
5177829aff revert twitter source change 2017-12-21 10:58:28 -08:00
r888888888
b5d72ae8d8 fixes #3422 2017-12-15 17:21:33 -08:00
evazion
4c39783d28 Fix #3424: /iqdb_queries.json fails for certain urls.
Fix the HTML page -> image URL download rewrite strategy failing for
https://lohas.nicoseiga.jp/thumb/${id}i URLs.
2017-12-15 10:16:06 -06:00
evazion
a14c492020 Fix #3396: Pixiv novel cover IDs are parsed incorrectly. 2017-11-21 13:03:03 -06:00
evazion
f8581c8113 Fix #3321: HTML tags not converted in artist commentaries. 2017-11-21 02:42:49 -06:00
r888888888
502f1298a9 fix unit tests 2017-11-20 16:30:07 -08:00
r888888888
9d5e4f969f fix source tests 2017-11-20 12:30:29 -08:00
evazion
5837b74de4 tumblr: fix exception when fetching source data.
Fixes an `undefined method `gsub' for nil:NilClass` exception when fetching source
data for this url:

* /uploads/new?url=https%3A%2F%2Fuiti-ukumo.tumblr.com%2Fpost%2F156157443158%2F2016-12

Bug introduced in 3c524aa5: `url` wasn't initialized due to `@url = url`
being removed from Sources::Site#initialize.
2017-11-18 15:44:09 -06:00
evazion
f896c24d37 tumblr: fail more gracefully if API key is not configured. 2017-11-18 13:53:11 -06:00
evazion
22c9cfcec5 Fix #3384: Fetch commentary not creating fully formed textile links 2017-11-18 13:02:50 -06:00
r888888888
3c524aa5e6 fix bug with source::site 2017-11-17 17:26:33 -08:00
r888888888
9a3824a87e fixes #3382 2017-11-17 16:18:59 -08:00
evazion
eeb41d2ffd twitter: get status id from referer url if not in direct url. 2017-11-16 13:29:58 -06:00
evazion
cbd68ac91b /source.json: do artist lookup and return matches in artists. 2017-11-14 23:28:25 -06:00
r888888888
d4008a5ddc fixes #3327 2017-10-09 16:32:05 -07:00
r888888888
449385f08f fixes #3313 2017-10-09 12:17:15 -07:00
lllusion3469
4c3733ce5a Keep auth_secure cookie from deviantart
Unlike this projects, all other DA scrapers I found[0] require that both
"auth" and "auth_secure" are set.
While the current approach, for some reason, seems to have worked fine in
the past, it has stopped functioning on some systems, sometimes
permanently (apparently sometimes only temporarily).
I don't have the means to find out the reason behind that, since all
configurations I've tried (mobile, VPN, and regular connection +
4 seperate accounts) resulted in me being effectively logged out and
unable to view mature works after deleting the "auth_secure" cookie.
The "if auth_cookie" part might not actually be necessary, its only
purpose is to prevent unintended breakage due to lack of information
on my part.

[0]: b662b46a44 (diff-aab4e263d591d1c7f23d015a2e540a32R161)
     cbaecf5507 (diff-9f9821e7d744874a2fe42e0a69e4b14aR33)
2017-08-30 20:47:59 +02:00
r888888888
bc69badadd better error handling of failed deviant art logins 2017-08-28 16:19:41 -07:00
r888888888
d124ff8128 fix case when fetching pixiv sources when illust_id param comes before mode 2017-08-23 17:50:33 -07:00
evazion
9b031a9c20 Fix #3266: Normalize fullwidth Unicode characters in translated tags. 2017-08-12 17:13:52 -05:00
Albert Yi
521927ae35 Merge pull request #3255 from evazion/fix-3254
Fix #3254, #3072: Some Twitter URLs upload wrong image.
2017-08-07 16:03:20 -07:00
evazion
c270954991 Fix #3258: Fetch source data broken for https://i.pximg.net/* ugoiras. 2017-08-06 17:23:29 -05:00
evazion
8b70e0099b twitter: fix to handle extended tweets (fix #3254, #3072). 2017-08-05 23:12:55 -05:00
Albert Yi
36a17d79f9 Merge pull request #3242 from evazion/fix-drop-whitecube
Drop Pixiv whitecube support (#2681, #3206).
2017-07-26 17:07:27 -07:00
evazion
80e8c9d174 strategies/pixiv.rb: removed unused methods (#3206).
The page count and the moniker are now obtained from the API.
2017-07-26 18:17:04 -05:00
evazion
7ae860a8b8 Drop Pixiv whitecube support (#2681, #3206). 2017-07-26 18:06:46 -05:00
r888888888
c80291c072 fix twitter:error:forbidden catch block 2017-07-21 18:12:46 -07:00
Albert Yi
9b982b3379 Merge pull request #3228 from r888888888/ccs
integrate with ccs
2017-07-21 18:00:05 -07:00
r888888888
c5a40628e6 integrate with ccs 2017-07-20 15:25:19 -07:00
evazion
93d44c6b84 Fix #3224: 403 Forbidden when logging in to DeviantArt.
Login fails if the User-Agent isn't set while POST'ing the login
credentials.
2017-07-19 20:25:27 -05:00
r888888888
97ce9cafac add httparty option config 2017-07-12 15:52:48 -07:00
r888888888
eb6c5e3af5 switch to httparty 2017-07-12 15:52:48 -07:00
evazion
cc8986641b Fix #3208: Fix translated tag suggestions for Pixiv.
* Only suggest the Danbooru tag with the same name if there is no
  matching wiki other name. Example: if we have the Pixiv tag `Fate` and
  the Danbooru tag `fate_(series)` with other name `fate`, suggest that,
  not the Danbooru tag `fate`.

* Don't suggest tags that are empty or whose wiki is deleted.

* Only split tags on "/" if there are no other matches, and only for Pixiv.

* For Pixiv, only include traditional media tags in tag list, not digital media (Photoshop, SAI).

* Add some tests.
2017-07-09 12:10:30 -05:00
evazion
71f84b10af tumblr: convert commentary to dtext.
* Convert Tumblr commentary to DText.
* Strip extraneous whitespace in links and blockquotes.
* Add newlines after block elements to ensure they're separated from
  subsequent blocks.
2017-07-01 11:15:48 -05:00
evazion
b880b07387 sources: factor out html-to-dtext code to DText.from_html. 2017-07-01 11:15:48 -05:00
evazion
118843b26a tumblr: fetch tags, commentary, and image urls. 2017-06-25 15:34:15 -05:00
evazion
294358b4a6 nijie: fetch image_urls for batch bookmarklet. 2017-06-20 16:29:31 -05:00
evazion
2d5fc191dd nijie: convert commentary to dtext. 2017-06-20 16:11:16 -05:00
evazion
25e7db860a nijie: fetch artist commentary. 2017-06-20 16:11:16 -05:00
evazion
14196f36d8 pawoo: convert commentary to dtext. 2017-06-19 17:10:02 -05:00
evazion
6edb5a807b pawoo: fetch tags and commentary. 2017-06-19 17:10:02 -05:00
evazion
1bfc0a7d8a pawoo: get status id from normalized url. 2017-06-19 17:09:57 -05:00
evazion
e91942dff3 pawoo: add source tests and docs. 2017-06-19 17:09:57 -05:00
Albert Yi
584b01457b Merge pull request #3172 from evazion/fix-twitter-tags
Fix #3171: Translate Twitter Hashtags to assist with tagging
2017-06-19 13:24:13 -07:00
Albert Yi
586a3186fd Merge pull request #3167 from evazion/feat-dtextify-pixiv-artcomm
Convert Pixiv commentaries to DText (fix #2458)
2017-06-19 13:22:31 -07:00
Type-kun
faa2d99160 Fix #3174 2017-06-19 18:22:58 +05:00
evazion
128e655aef twitter: fetch hashtags for translated tags (fixes #3171). 2017-06-17 16:15:23 -05:00