jaculabilis/danbooru
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
6,479 Commits 1 Branch 0 Tags
118fd3baf4e04ba54bddb15aedded2e456a3b3bb
Commit Graph

222 Commits

Author SHA1 Message Date
evazion
a14c492020 Fix #3396: Pixiv novel cover IDs are parsed incorrectly. 2017-11-21 13:03:03 -06:00
evazion
f8581c8113 Fix #3321: HTML tags not converted in artist commentaries. 2017-11-21 02:42:49 -06:00
r888888888
502f1298a9 fix unit tests 2017-11-20 16:30:07 -08:00
r888888888
9d5e4f969f fix source tests 2017-11-20 12:30:29 -08:00
evazion
5837b74de4 tumblr: fix exception when fetching source data. 
Fixes an `undefined method `gsub' for nil:NilClass` exception when fetching source
data for this url:

* /uploads/new?url=https%3A%2F%2Fuiti-ukumo.tumblr.com%2Fpost%2F156157443158%2F2016-12

Bug introduced in 3c524aa5: `url` wasn't initialized due to `@url = url`
being removed from Sources::Site#initialize.
 2017-11-18 15:44:09 -06:00
evazion
f896c24d37 tumblr: fail more gracefully if API key is not configured. 2017-11-18 13:53:11 -06:00
evazion
22c9cfcec5 Fix #3384: Fetch commentary not creating fully formed textile links 2017-11-18 13:02:50 -06:00
r888888888
3c524aa5e6 fix bug with source::site 2017-11-17 17:26:33 -08:00
r888888888
9a3824a87e fixes #3382 2017-11-17 16:18:59 -08:00
evazion
eeb41d2ffd twitter: get status id from referer url if not in direct url. 2017-11-16 13:29:58 -06:00
evazion
cbd68ac91b /source.json: do artist lookup and return matches in artists. 2017-11-14 23:28:25 -06:00
r888888888
d4008a5ddc fixes #3327 2017-10-09 16:32:05 -07:00
r888888888
449385f08f fixes #3313 2017-10-09 12:17:15 -07:00
lllusion3469
4c3733ce5a Keep auth_secure cookie from deviantart 
Unlike this projects, all other DA scrapers I found[0] require that both
"auth" and "auth_secure" are set.
While the current approach, for some reason, seems to have worked fine in
the past, it has stopped functioning on some systems, sometimes
permanently (apparently sometimes only temporarily).
I don't have the means to find out the reason behind that, since all
configurations I've tried (mobile, VPN, and regular connection +
4 seperate accounts) resulted in me being effectively logged out and
unable to view mature works after deleting the "auth_secure" cookie.
The "if auth_cookie" part might not actually be necessary, its only
purpose is to prevent unintended breakage due to lack of information
on my part.

[0]: b662b46a44 (diff-aab4e263d591d1c7f23d015a2e540a32R161)
     cbaecf5507 (diff-9f9821e7d744874a2fe42e0a69e4b14aR33)
 2017-08-30 20:47:59 +02:00
r888888888
bc69badadd better error handling of failed deviant art logins 2017-08-28 16:19:41 -07:00
r888888888
d124ff8128 fix case when fetching pixiv sources when illust_id param comes before mode 2017-08-23 17:50:33 -07:00
evazion
9b031a9c20 Fix #3266: Normalize fullwidth Unicode characters in translated tags. 2017-08-12 17:13:52 -05:00
Albert Yi
521927ae35 Merge pull request #3255 from evazion/fix-3254 
Fix #3254, #3072: Some Twitter URLs upload wrong image.
 2017-08-07 16:03:20 -07:00
evazion
c270954991 Fix #3258: Fetch source data broken for https://i.pximg.net/* ugoiras. 2017-08-06 17:23:29 -05:00
evazion
8b70e0099b twitter: fix to handle extended tweets (fix #3254, #3072). 2017-08-05 23:12:55 -05:00
Albert Yi
36a17d79f9 Merge pull request #3242 from evazion/fix-drop-whitecube 
Drop Pixiv whitecube support (#2681, #3206).
 2017-07-26 17:07:27 -07:00
evazion
80e8c9d174 strategies/pixiv.rb: removed unused methods (#3206). 
The page count and the moniker are now obtained from the API.
 2017-07-26 18:17:04 -05:00
evazion
7ae860a8b8 Drop Pixiv whitecube support (#2681, #3206). 2017-07-26 18:06:46 -05:00
r888888888
c80291c072 fix twitter:error:forbidden catch block 2017-07-21 18:12:46 -07:00
Albert Yi
9b982b3379 Merge pull request #3228 from r888888888/ccs 
integrate with ccs
 2017-07-21 18:00:05 -07:00
r888888888
c5a40628e6 integrate with ccs 2017-07-20 15:25:19 -07:00
evazion
93d44c6b84 Fix #3224: 403 Forbidden when logging in to DeviantArt. 
Login fails if the User-Agent isn't set while POST'ing the login
credentials.
 2017-07-19 20:25:27 -05:00
r888888888
97ce9cafac add httparty option config 2017-07-12 15:52:48 -07:00
r888888888
eb6c5e3af5 switch to httparty 2017-07-12 15:52:48 -07:00
evazion
cc8986641b Fix #3208: Fix translated tag suggestions for Pixiv. 
* Only suggest the Danbooru tag with the same name if there is no
  matching wiki other name. Example: if we have the Pixiv tag `Fate` and
  the Danbooru tag `fate_(series)` with other name `fate`, suggest that,
  not the Danbooru tag `fate`.

* Don't suggest tags that are empty or whose wiki is deleted.

* Only split tags on "/" if there are no other matches, and only for Pixiv.

* For Pixiv, only include traditional media tags in tag list, not digital media (Photoshop, SAI).

* Add some tests.
 2017-07-09 12:10:30 -05:00
evazion
71f84b10af tumblr: convert commentary to dtext. 
* Convert Tumblr commentary to DText.
* Strip extraneous whitespace in links and blockquotes.
* Add newlines after block elements to ensure they're separated from
  subsequent blocks.
 2017-07-01 11:15:48 -05:00
evazion
b880b07387 sources: factor out html-to-dtext code to DText.from_html. 2017-07-01 11:15:48 -05:00
evazion
118843b26a tumblr: fetch tags, commentary, and image urls. 2017-06-25 15:34:15 -05:00
evazion
294358b4a6 nijie: fetch image_urls for batch bookmarklet. 2017-06-20 16:29:31 -05:00
evazion
2d5fc191dd nijie: convert commentary to dtext. 2017-06-20 16:11:16 -05:00
evazion
25e7db860a nijie: fetch artist commentary. 2017-06-20 16:11:16 -05:00
evazion
14196f36d8 pawoo: convert commentary to dtext. 2017-06-19 17:10:02 -05:00
evazion
6edb5a807b pawoo: fetch tags and commentary. 2017-06-19 17:10:02 -05:00
evazion
1bfc0a7d8a pawoo: get status id from normalized url. 2017-06-19 17:09:57 -05:00
evazion
e91942dff3 pawoo: add source tests and docs. 2017-06-19 17:09:57 -05:00
Albert Yi
584b01457b Merge pull request #3172 from evazion/fix-twitter-tags 
Fix #3171: Translate Twitter Hashtags to assist with tagging
 2017-06-19 13:24:13 -07:00
Albert Yi
586a3186fd Merge pull request #3167 from evazion/feat-dtextify-pixiv-artcomm 
Convert Pixiv commentaries to DText (fix #2458)
 2017-06-19 13:22:31 -07:00
Type-kun
faa2d99160 Fix #3174 2017-06-19 18:22:58 +05:00
evazion
128e655aef twitter: fetch hashtags for translated tags (fixes #3171). 2017-06-17 16:15:23 -05:00
evazion
94e48aa6f3 pixiv: convert commentary to dtext. 
Convert pixiv links to dtext in this manner:

* https://www.pixiv.net/member_illust.php?mode=medium&illust_id=1234
  => pixiv #1234 "»":[/posts?tags=pixiv:1234]

* https://www.pixiv.net/member.php?id=1234
  => "user/1234":[https://www.pixiv.net/member.php?id=1234] "»":[/artists?search[url_matches]=https://www.pixiv.net/member.php?id=1234]
 2017-06-16 18:42:37 -05:00
evazion
5dd3151d5b twitter: convert commentary to dtext. 
* Convert hashtags and mentions to dtext links.
* Replace http://t.co urls to the actual url.
* Strip the http://t.co url linking to the tweet itself.
 2017-06-16 12:56:55 -05:00
evazion
9909ff20cd twitter: memoize api response. 2017-06-16 12:56:55 -05:00
r888888888
6174d0eef2 normalize nijie popup urls (fixes #3153) 2017-06-14 12:26:13 -07:00
evazion
f3f55daea6 Fix #3138: Support DeviantArt login-only works. 2017-06-10 10:34:51 -05:00
evazion
82c7537270 deviantart: force commentary encoding to utf-8. 2017-06-09 13:19:24 -05:00