Commit Graph

72 Commits

Author SHA1 Message Date
Albert Yi
b0f2b1e8dd Merge pull request #2782 from evazion/fix-2779
Fix #2779: Erroneous pixiv_id parsed for novel/background/profile images
2016-12-05 11:44:38 -08:00
Albert Yi
bd9939a50b some additional documentation about rewrite strategies 2016-12-05 11:43:18 -08:00
evazion
838d6f6eea Fix #2460: Uploading pixiv user's background image throws error.
Fix exception when uploading these types of URLs:

* http://img12.pixiv.net/profile/rapattu/119950.jpg
* http://i2.pixiv.net/img130/profile/minono_aki/8733472.jpg
* http://i2.pixiv.net/img20/img/a-park/novel/3607898.jpg
* http://i1.pixiv.net/novel-cover-original/img/2016/11/03/20/10/58/7436075_f75af69f3eacd1656d3733c72aa959cf.jpg
* http://i1.pixiv.net/background/img/2016/05/17/12/05/48/2074388_d4ac52034f7ca0af3e083d59fde7e97f.jpg

Skip rewriting sources that don't have illust IDs to avoid this
exception. Also fix `tags` so that it doesn't blow up when fetching
source data for these types of sources.
2016-12-04 02:04:44 -06:00
evazion
3eafc5ae6a Don't clobber image board sources. 2016-11-19 02:29:33 -06:00
evazion
f94dfbf17e Fix bookmarklet for whitecube URLs.
Make the bookmarklet work for URLs like this:

* http://www.pixiv.net/whitecube/user/607414/illust/59510800
* http://www.pixiv.net/whitecube/user/772090/illust/59595705
2016-10-22 16:11:44 -05:00
r888888888
4fa2741ed2 include whitecube source/rewrite strategies 2016-09-28 11:24:54 -07:00
Toks
98ef1816ba Rewrite nijie html view page to image url 2015-12-22 12:29:46 -05:00
Toks
aa212e3616 Fix error when uploading old deviantart url format 2015-09-21 10:47:24 -04:00
Toks
2359f41441 Reorganize rewrite strats slightly 2015-09-19 10:26:40 -04:00
Toks
a072fae19d Fix "EOFError - end of file reached" for https tumblr uploads 2015-09-09 10:41:59 -04:00
Toks
12be880fa9 Support fetching artist commentary from deviantart 2015-09-03 20:03:56 -04:00
r888888888
f85d5a5570 addresses #2498: revised implementation 2015-08-18 12:45:10 -07:00
r888888888
407523b04c addresses #2498: Protect source downloader against server-side request forgery attacks 2015-08-18 12:26:26 -07:00
Toks
e165cb0ab2 fixes #2491 2015-08-15 17:12:00 -04:00
Toks
6ea556944b Add support for uploading from seiga /o/ pages 2015-07-07 18:00:47 -04:00
Toks
462e1ed324 Implement #2395 for nijie twitter and tumblr 2015-06-13 11:26:47 -04:00
Toks
7360fc4633 fix typo 2015-06-13 10:56:27 -04:00
r888888888
2ddc93723c fixes #2327 2015-06-02 19:20:09 -07:00
Toks
367b6495ae rename 2015-05-28 02:25:25 -04:00
Toks
9481f09b93 Support rewriting direct twitter url -> full size url
If the user gives us the direct image url the twitter api is useless
since it doesn't have the status id. So just rewrite the url itself.
2015-05-28 02:20:45 -04:00
Toks
bb4267af7d remove debug 2015-05-13 18:32:02 -04:00
r888888888
ac6d1fa498 bug fix with pixiv api 2015-05-11 15:53:45 -07:00
Toks
04e6aa7474 #1866 update for new deviantart urls 2015-03-13 14:12:17 -04:00
r888888888
b353b21c53 fixes #2353 2015-03-05 17:58:00 -08:00
r888888888
f12fb40e3e implements #2350 2015-02-20 20:04:23 -08:00
r888888888
fc30b9d606 rely on twitter api for source calls 2015-02-09 17:50:19 -08:00
r888888888
e57cad7b0a add new uploads/batch endpoint for handling twitter galleries 2015-02-09 16:29:00 -08:00
Toks
712a5844b3 Update for new twitpic link format
related #1866
2015-01-05 21:39:19 -05:00
Toks
6dca8d5915 Fix thumbnail rewrite for png images 2014-12-22 11:35:36 -05:00
Toks
4231fac4cb #2328: fix image board src rewrite 2014-12-18 11:14:25 -05:00
r888888888
43ee2875bf potential fix for #2328 2014-12-17 14:18:25 -08:00
r888888888
4fcb1d2bbc support for twitter downloads 2014-12-05 14:19:36 -08:00
r888888888
cdc6467a91 fix regexp 2014-12-03 12:48:07 -08:00
r888888888
97a71fe5a0 fixes #2298 2014-12-03 12:46:05 -08:00
r888888888
16f9a61d63 fixes #2299 2014-10-29 15:14:17 -07:00
r888888888
981e6dab67 no longer require presence of ugoira tag to process ugoira posts 2014-10-23 16:34:31 -07:00
r888888888
69564c4bff fix async processing for ugoira 2014-10-22 22:48:20 -07:00
evazion
38cb35ffbd Only fetch ugoira data for ugoira URLs.
Don't fetch ugoira data for http://i1.pixiv.net/img35/img/kinokoyarou/46165361.png URLs.
2014-10-19 02:30:03 -07:00
evazion
c0a803e3a6 Allow ugoira to be uploaded directly from the zip URL.
This refactors things such that

* Fetching the ugoira frame data happens at the end of rewrite, not
  rewrite_html_pages. This way uploading zip URLs should work.
* The source data is cached in an instance variable in case it
  was indeed fetched during rewrite_html_pages. This way it doesn't get
  fetched a second time.
2014-10-19 02:30:03 -07:00
Toks
d47327be1d Fix argument error 2014-10-19 02:30:02 -07:00
r888888888
4c73fb9f79 add ugoira support in view 2014-10-19 02:30:02 -07:00
r888888888
3bb06c2be4 integrate ugoiras into zip+webm+preview 2014-10-19 02:30:02 -07:00
r888888888
0a61aac231 refactor 2014-10-19 02:30:02 -07:00
r888888888
fb2219d4ac integrate ugoira converted into upload flow 2014-10-19 02:30:01 -07:00
evazion
964b5efcd3 Rewrite Pixiv small/medium images to full size images.
There are two kinds of thumbnails that need to be rewritten. First case:
new /img-master/ URLs need to be rewritten to /img-original/ URLs like this:

    http://i2.pixiv.net/c/600x600/img-master/img/2014/10/04/03/59/52/46337015_p0_master1200.jpg
    => http://i2.pixiv.net/img-original/img/2014/10/04/03/59/52/46337015_p0.png

This is what `rewrite_new_medium_images` does. In order to do this, it
has to use the Pixiv API to get the correct file extension.

Second case: Old small/medium size URLs need to be rewritten to full
size URLs like this:

    http://i2.pixiv.net/img18/img/evazion/14901720_m.png
    => http://i2.pixiv.net/img18/img/evazion/14901720.png

But when the medium size URL is actually for a manga image, it needs to be
rewritten to the big manga URL instead:

    http://i2.pixiv.net/img04/img/syounen_no_uta/46170939_m.jpg
    => http://i2.pixiv.net/img04/img/syounen_no_uta/46170939_big_p0.jpg

But we can't tell whether it's a manga image from the URL, so we have to
use the manga page count from either the HTML page or the API to
determine whether it's part of a manga gallery.

So in order to make this work, `rewrite_old_small_and_medium_images`
takes an `is_manga` flag. `Sources::Strategies::Pixiv#get` gets the
page count from the HTML and passes the `is_manga` flag on down through
the call chain until `rewrite_old_small_and_medium_images` gets it.

When `rewrite_old_small_and_medium_images` is called from
`Downloads::Strategies::Pixiv#rewrite_thumbnails`, the `is_manga` flag
isn't passed in because we didn't scrape the HTML. This causes
`rewrite_old_small_and_medium_images` to look it up in the API instead.
2014-10-04 12:45:37 -05:00
evazion
daf196bd0e Rewrite old Pixiv small manga images correctly.
This prevents incorrectly rewriting these new Pixiv URLs:

* http://i1.pixiv.net/c/600x600/img-master/img/2014/09/24/23/25/08/46168376_p0_master1200.jpg
* http://i1.pixiv.net/img-original/img/2014/09/25/23/09/29/46183440_p0.jpg
2014-10-04 12:45:37 -05:00
Toks
4b9605075b Rewrite tumblr cdn urls 2014-10-03 14:36:37 -04:00
Toks
d499bb4d04 Remove tumblr 1280w -> 500h source rewriting
It was thought that 500h was always the biggest size when it exists, but
it seems it can actually be either bigger or smaller than 1280w.
2014-10-03 14:21:53 -04:00
Toks
3cac0458df fix regex 2014-07-16 21:10:54 -04:00
Toks
364feba64a Rewrite yandere/konachan jpeg samples 2014-07-16 20:53:33 -04:00