jaculabilis/danbooru
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
3,710 Commits 1 Branch 0 Tags
f22cb52dde667ddc6b6a7fb08fd04cdab41cdaa3
Commit Graph
550 Commits
Author SHA1 Message Date
evazion ec0f226f46 Make the artist finder work with new Pixiv URLs. 2014-10-04 12:45:37 -05:00
evazion 964b5efcd3 Rewrite Pixiv small/medium images to full size images. 
There are two kinds of thumbnails that need to be rewritten. First case:
new /img-master/ URLs need to be rewritten to /img-original/ URLs like this:

    http://i2.pixiv.net/c/600x600/img-master/img/2014/10/04/03/59/52/46337015_p0_master1200.jpg
    => http://i2.pixiv.net/img-original/img/2014/10/04/03/59/52/46337015_p0.png

This is what `rewrite_new_medium_images` does. In order to do this, it
has to use the Pixiv API to get the correct file extension.

Second case: Old small/medium size URLs need to be rewritten to full
size URLs like this:

    http://i2.pixiv.net/img18/img/evazion/14901720_m.png
    => http://i2.pixiv.net/img18/img/evazion/14901720.png

But when the medium size URL is actually for a manga image, it needs to be
rewritten to the big manga URL instead:

    http://i2.pixiv.net/img04/img/syounen_no_uta/46170939_m.jpg
    => http://i2.pixiv.net/img04/img/syounen_no_uta/46170939_big_p0.jpg

But we can't tell whether it's a manga image from the URL, so we have to
use the manga page count from either the HTML page or the API to
determine whether it's part of a manga gallery.

So in order to make this work, `rewrite_old_small_and_medium_images`
takes an `is_manga` flag. `Sources::Strategies::Pixiv#get` gets the
page count from the HTML and passes the `is_manga` flag on down through
the call chain until `rewrite_old_small_and_medium_images` gets it.

When `rewrite_old_small_and_medium_images` is called from
`Downloads::Strategies::Pixiv#rewrite_thumbnails`, the `is_manga` flag
isn't passed in because we didn't scrape the HTML. This causes
`rewrite_old_small_and_medium_images` to look it up in the API instead.
 2014-10-04 12:45:37 -05:00
evazion daf196bd0e Rewrite old Pixiv small manga images correctly. 
This prevents incorrectly rewriting these new Pixiv URLs:

* http://i1.pixiv.net/c/600x600/img-master/img/2014/09/24/23/25/08/46168376_p0_master1200.jpg
* http://i1.pixiv.net/img-original/img/2014/09/25/23/09/29/46183440_p0.jpg
 2014-10-04 12:45:37 -05:00
evazion 36a78361d7 Normalize URLs to the mode=medium page correctly. 
This handles a few new cases that weren't handled correctly previously.

* http://i1.pixiv.net/img-zip-ugoira/img/2014/10/03/17/29/16/46323924_ugoira1920x1080.zip
* http://i1.pixiv.net/c/600x600/img-master/img/2014/10/02/13/51/23/46304396_p0_master1200.jpg
* http://www.pixiv.net/member_illust.php?mode=manga&illust_id=18557054
* http://www.pixiv.net/member_illust.php?mode=manga_big&illust_id=18557054&page=1
* http://www.pixiv.net/i/18557054
 2014-10-04 12:45:36 -05:00
evazion f889dbf10f Add get_metadata_from_spapi! 2014-10-04 12:45:36 -05:00
evazion 7f98b370ec Fix scraping the Pixiv artist username. 
The artist's username is no longer contained in the image thumbnail URL on the
HTML page. Get it from the Feed link instead.
 2014-10-04 12:45:36 -05:00
evazion 74c116ffb7 Fix for scraping the manga page count. 
The string for the page count has changed. It now looks like "複数枚投稿 3P"
on all Pixiv posts I've checked.
 2014-10-04 12:45:36 -05:00
Toks 4b9605075b Rewrite tumblr cdn urls 2014-10-03 14:36:37 -04:00
Toks d499bb4d04 Remove tumblr 1280w -> 500h source rewriting 
It was thought that 500h was always the biggest size when it exists, but
it seems it can actually be either bigger or smaller than 1280w.
 2014-10-03 14:21:53 -04:00
Toks 6dce66f33d Cache pixiv session 2014-09-29 01:38:53 -04:00
Toks fd290076fe fixes #2241 2014-09-24 13:28:56 -04:00
Toks 4db0c3c0e5 fixes #2249 2014-09-24 13:11:54 -04:00
r888888888 267df896c6 fixes #2245 2014-08-25 16:41:27 -07:00
r888888888 2b96040a30 fixes #2191 2014-08-12 17:03:16 -07:00
Toks ed6ed14a79 #2217 fix ambiguous table reference 2014-08-03 16:38:05 -04:00
Toks b22a77d4c5 improve #1678 
Remove blank &page= parameter when page isn't specified, and indicate
page in the link's text when it is.
 2014-08-01 17:59:59 -04:00
Toks 2b3909496a Fix detection of tag wildcard searches 2014-07-30 16:29:55 -04:00
r888888888 6772566665 fixes #2166 2014-07-23 15:15:47 -07:00
r888888888 44b59ab18d fixes #2203 2014-07-22 15:43:28 -07:00
r888888888 b025b7379b fix promotion report 2014-07-18 16:37:35 -07:00
r888888888 bb402f5a27 fixes #2197 2014-07-17 16:34:54 -07:00
Toks 3cac0458df fix regex 2014-07-16 21:10:54 -04:00
Toks 364feba64a Rewrite yandere/konachan jpeg samples 2014-07-16 20:53:33 -04:00
Toks 5aca6aa7c9 Fix pixiv gallery page counts 2014-07-16 11:58:43 -04:00
Toks a896e8704b Merge branch 'master' of https://github.com/r888888888/danbooru 2014-07-11 02:26:37 -04:00
Toks 302fa7efcd Rewrite pixiv cdn urls 2014-07-11 02:26:19 -04:00
r888888888 e86c8ceb2a add deletion column to promotion report 2014-07-08 10:21:07 -07:00
Toks bc912ecd00 #2217 fix ambiguous table references 2014-07-06 15:16:52 -04:00
Toks 772003a55a fixes #2217 2014-07-06 12:03:48 -04:00
Toks 224da8a7da Prevent pixiv downloader from returning wrong image 2014-06-25 11:14:08 -04:00
r888888888 dae7c615e9 fixes #2206 2014-06-24 11:07:39 -07:00
Toks 3cdb9fe1c2 Support new 4chan url format 2014-06-22 20:02:51 -04:00
r888888888 2aa9288d52 fixes #1678 2014-06-20 17:36:19 -07:00
r888888888 f65691b5e6 fix tests 2014-06-18 16:51:55 -07:00
r888888888 f02f72fac1 implement saved searches, move user boolean settings to bitprefs 2014-06-18 16:16:27 -07:00
Toks 500f7a7812 Potential fix for #1985 2014-06-17 17:49:41 -04:00
Toks 3103b97c8b fixes #2198 2014-06-17 13:19:40 -04:00
Toks 15d4c90688 Fix bug where importing mass updates would not work 2014-06-17 02:34:54 -04:00
Toks 9621ec7dac Support translating Pixiv "x users iri" tags 2014-06-16 14:20:28 -04:00
Toks be28a8e624 Fix Seiga sample/thumbnail rewriting 2014-06-13 16:59:08 -04:00
Toks 3230ab8781 Add warning when Pixiv post is a gallery of multiple images 2014-06-13 16:33:38 -04:00
Toks cb1f00b448 Fix bug where Pixiv /img-inf/ urls were incorrectly normalized 2014-06-13 16:18:25 -04:00
r888888888 075f11e71d fixes #2185 2014-06-12 21:45:17 -07:00
r888888888 9aa1f65eb5 add shortcut for currentuser.admin 2014-06-11 17:47:07 -07:00
r888888888 e64dac0b5d more lenient regexp for #2185 2014-06-11 16:56:45 -07:00
r888888888 554007ce91 fixes #2185 2014-06-11 16:53:24 -07:00
Toks 4a45d1e6bb Potential fix for #2124 2014-06-06 13:43:27 -04:00
Toks 7ca7ac2709 #1866: Support Nijie source data getting 2014-06-03 18:42:24 -04:00
r888888888 331377a32b implements #982 2014-06-03 15:23:16 -07:00
Toks e1f09e01b0 fixes #2173 2014-06-01 13:39:19 -04:00