Commit Graph

61 Commits

Author SHA1 Message Date
evazion
ec0f226f46 Make the artist finder work with new Pixiv URLs. 2014-10-04 12:45:37 -05:00
evazion
964b5efcd3 Rewrite Pixiv small/medium images to full size images.
There are two kinds of thumbnails that need to be rewritten. First case:
new /img-master/ URLs need to be rewritten to /img-original/ URLs like this:

    http://i2.pixiv.net/c/600x600/img-master/img/2014/10/04/03/59/52/46337015_p0_master1200.jpg
    => http://i2.pixiv.net/img-original/img/2014/10/04/03/59/52/46337015_p0.png

This is what `rewrite_new_medium_images` does. In order to do this, it
has to use the Pixiv API to get the correct file extension.

Second case: Old small/medium size URLs need to be rewritten to full
size URLs like this:

    http://i2.pixiv.net/img18/img/evazion/14901720_m.png
    => http://i2.pixiv.net/img18/img/evazion/14901720.png

But when the medium size URL is actually for a manga image, it needs to be
rewritten to the big manga URL instead:

    http://i2.pixiv.net/img04/img/syounen_no_uta/46170939_m.jpg
    => http://i2.pixiv.net/img04/img/syounen_no_uta/46170939_big_p0.jpg

But we can't tell whether it's a manga image from the URL, so we have to
use the manga page count from either the HTML page or the API to
determine whether it's part of a manga gallery.

So in order to make this work, `rewrite_old_small_and_medium_images`
takes an `is_manga` flag. `Sources::Strategies::Pixiv#get` gets the
page count from the HTML and passes the `is_manga` flag on down through
the call chain until `rewrite_old_small_and_medium_images` gets it.

When `rewrite_old_small_and_medium_images` is called from
`Downloads::Strategies::Pixiv#rewrite_thumbnails`, the `is_manga` flag
isn't passed in because we didn't scrape the HTML. This causes
`rewrite_old_small_and_medium_images` to look it up in the API instead.
2014-10-04 12:45:37 -05:00
evazion
36a78361d7 Normalize URLs to the mode=medium page correctly.
This handles a few new cases that weren't handled correctly previously.

* http://i1.pixiv.net/img-zip-ugoira/img/2014/10/03/17/29/16/46323924_ugoira1920x1080.zip
* http://i1.pixiv.net/c/600x600/img-master/img/2014/10/02/13/51/23/46304396_p0_master1200.jpg
* http://www.pixiv.net/member_illust.php?mode=manga&illust_id=18557054
* http://www.pixiv.net/member_illust.php?mode=manga_big&illust_id=18557054&page=1
* http://www.pixiv.net/i/18557054
2014-10-04 12:45:36 -05:00
evazion
f889dbf10f Add get_metadata_from_spapi! 2014-10-04 12:45:36 -05:00
evazion
7f98b370ec Fix scraping the Pixiv artist username.
The artist's username is no longer contained in the image thumbnail URL on the
HTML page. Get it from the Feed link instead.
2014-10-04 12:45:36 -05:00
evazion
74c116ffb7 Fix for scraping the manga page count.
The string for the page count has changed. It now looks like "複数枚投稿 3P"
on all Pixiv posts I've checked.
2014-10-04 12:45:36 -05:00
Toks
6dce66f33d Cache pixiv session 2014-09-29 01:38:53 -04:00
Toks
5aca6aa7c9 Fix pixiv gallery page counts 2014-07-16 11:58:43 -04:00
Toks
224da8a7da Prevent pixiv downloader from returning wrong image 2014-06-25 11:14:08 -04:00
Toks
9621ec7dac Support translating Pixiv "x users iri" tags 2014-06-16 14:20:28 -04:00
Toks
be28a8e624 Fix Seiga sample/thumbnail rewriting 2014-06-13 16:59:08 -04:00
Toks
3230ab8781 Add warning when Pixiv post is a gallery of multiple images 2014-06-13 16:33:38 -04:00
Toks
7ca7ac2709 #1866: Support Nijie source data getting 2014-06-03 18:42:24 -04:00
Toks
d092ea0094 fixes #1207 2014-05-29 23:43:19 -04:00
Toks
eb81f06eb2 merge translated tags branch 2014-05-29 23:11:34 -04:00
Toks
4f8b455830 fixes #2168 2014-05-29 18:46:09 -04:00
Toks
5f70768962 #1866: Support HTTPS urls 2014-05-23 14:15:33 -04:00
Toks
38c0e01f9b Support referrer matching for seiga and da 2014-05-23 14:15:23 -04:00
Toks
b18bb73f4b Implentation for #2141 2014-05-22 20:07:15 -04:00
Toks
a3d120c632 #1866: Support HTTPS urls 2014-05-15 23:35:57 -04:00
Toks
0a75402cc7 Support referrer matching for seiga and da 2014-05-08 20:25:11 -04:00
Toks
47f56cd19d #1866: Fix deviantart regex again and support alternate url style 2014-05-04 15:54:11 -04:00
Toks
bb07dc429b Seiga: fix source uploads still not working in some cases 2014-04-30 15:18:53 -04:00
Toks
884be2b711 Seiga: fix source uploads not working 2014-04-30 14:40:21 -04:00
Toks
281c7e4bf7 Seiga: fix getting tags 2014-04-30 14:32:14 -04:00
Toks
899fd8f71f Don't instantly make a request to get info when using bookmarklet
Currently this is done twice: once when the page first loads (although
this one isn't used) and then a second time asynchronously with
javascript (which is used). This commit removes the first one, improving
upload page load time.
2014-04-30 14:28:07 -04:00
Toks
ce2bcc4570 Seiga: support alternate type of direct link url 2014-04-30 12:31:41 -04:00
Toks
b559f11c99 Seiga: fix getting artist name 2014-04-30 12:29:35 -04:00
Toks
0507064004 #1866: Add nico seiga support and fix various seiga bugs
* Support rewriting source when user uploads from a thumbnail url or
html page url
* Fix bug where site did not log in correctly
* Fix bug where the image url couldn't be extracted from the page if the
image was rated as adults only on seiga
* Normalize direct image url to html page url so tags, etc., can be
extracted
2014-04-29 11:46:08 -04:00
Toks
0cba44e3bf Fix deviantart regex bugs 2014-04-28 15:49:50 -04:00
Toks
6fbd373873 #1866: Support deviantart source rewriting
* html work page -> full image
* thumbnail -> full image
2013-11-26 22:43:23 -05:00
r888888888
e2571e74cc refactored nico seiga sources 2013-08-07 18:12:16 -07:00
小太
cba839ba76 Kill trailing whitespace in ruby files 2013-03-19 23:10:10 +11:00
albert
d8201b7c6d fixes #868 2013-03-14 17:36:30 -04:00
albert
43177cbad8 fixes #827 2013-03-10 16:56:04 -04:00
albert
8fb20c6b3a fixed nico seiga sources 2013-02-19 23:09:07 -05:00
albert
7f11fb4583 fix for artist search 2013-02-19 21:41:35 -05:00
albert
0a0d25e0a9 fix for pixiv urls 2013-02-19 12:07:40 -05:00
albert
eb1ba8f708 fix tests 2013-02-17 21:32:07 -05:00
albert
7e0e961656 pixiv source fix 2013-02-16 23:26:28 -05:00
albert
f5f760117e pixiv source fix 2013-02-16 23:25:33 -05:00
albert
3cb84d93bd fix 2013-02-16 23:15:17 -05:00
albert
e1d78185a0 fix 2013-02-16 23:10:14 -05:00
albert
b6e9e5b41f fix pixiv source 2013-01-06 15:59:31 -05:00
albert
f44ba8fe9f fix pixiv source 2012-12-26 15:11:21 -05:00
albert
17881068e1 * Removed Pixa/Tinami sources
* Upgraded to Rails 3.2.3
* Fixed tests
2012-06-01 19:22:58 -04:00
albert
c14f020ce2 * Fix for Pixiv changes
* Fix for artist/wiki pages for -names
2012-02-08 12:44:17 -05:00
albert
87bfa04e65 fixed pixa source 2011-10-07 14:25:55 -04:00
albert
2287bc8d61 fixed pixiv sources 2011-09-30 13:27:22 -04:00
albert
895ae18160 code cleanup 2011-09-29 16:16:03 -04:00