jaculabilis/danbooru
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
12,019 Commits 1 Branch 0 Tags
902cd0bbdf90337cc26935270e08ae922ea05d2e
Commit Graph

123 Commits

Author SHA1 Message Date
evazion
902cd0bbdf sources: add 4chan.org upload support. 
Add upload support for 4chan.org. You can upload either individual posts or entire threads at a time.
 2022-11-18 01:23:15 -06:00
evazion
a747911df8 foundation: fix foundation.app/@name/~/$id type URLs. 
Fix URLs like this not working:

    https://foundation.app/@mochiiimo/~/97376

The correct URL is now:

    https://foundation.app/@mochiiimo/foundation/97376
 2022-11-16 22:26:01 -06:00
evazion
e12ce9b472 Fix #5343: Add icons for websites linked to at least 9 artists. 2022-11-12 19:49:20 -06:00
evazion
4c3b988d15 sources: fix site icons. 2022-11-11 14:03:06 -06:00
evazion
0fd95375fa sources: add more site icons. 
Add site icons for the following sites:

* 4chan
* Catbox
* Danbooru
* Drawcrowd
* E-Hentai
* Enty
* Gelbooru
* Hitomi
* Imgur
* Joyreactor
* Kemono Party
* Konachan
* Luscious.net
* Mega.nz
* Monappy
* Mihoyo
* Overdoll
* Pinterest
* Rule34.us
* Sankaku Complex
* Steam
* Webmshare
* Yande.re
* Zerochan

These are mainly used for source links on the https://danbooru.donmai.us/post_events?search[category]=Replacement page.
 2022-11-11 00:54:50 -06:00
evazion
15f611ca8b Add Rule34.us support. 2022-11-10 12:53:06 -06:00
evazion
ec87cffe02 Add Rule34.xxx support. 2022-11-10 00:39:52 -06:00
evazion
0bad53e6ac Add TBIB.org support. 2022-11-09 16:49:20 -06:00
evazion
5b4856273f Add safebooru.org support. 
Refactor the Gelbooru source extractor to support Safebooru.org as well.
 2022-11-09 15:44:04 -06:00
nonamethanks
35bfcbc3bd Newgrounds: support video uploads 2022-11-09 15:01:28 +01:00
evazion
53d4052387 Merge pull request #5329 from nonamethanks/feat-bilibili 
Add bilibili support
 2022-11-09 01:17:18 -06:00
evazion
153d5f3e67 moebooru: support md5-based post URLs. 
Support URLs like these:

* https://yande.re/post/show?md5=2c95b8975b73744da2bcbed9619c1d59
* https://konachan.com/post/show?md5=955aa45f3b452b415509b47dcc9475ac

This makes scraping images from these sites by md5 easier.
 2022-11-08 21:40:16 -06:00
nonamethanks
0a78a6188d Reddit: support user galleries and embedded images 2022-11-06 15:04:05 +01:00
evazion
af9e570419 skeb: parse www.skeb.jp URLs. 2022-11-05 19:09:56 -05:00
evazion
ab08ad24e6 baraag: parse https://baraag.net/web/@{name} URLs. 2022-11-05 19:09:56 -05:00
evazion
fcb68048e3 fanbox: parse https://pixiv.net/fanbox URLs. 2022-11-05 19:09:56 -05:00
evazion
f5b003fc94 furaffinity: parse https://www.furaffinity.net/stats/{name} URLs. 2022-11-05 19:09:56 -05:00
nonamethanks
758b14ad46 Remove superfluous site name definitions and update artist url order 2022-11-05 20:03:07 +01:00
nonamethanks
fc4d620002 Add bilibili support 2022-11-04 12:11:08 +01:00
nonamethanks
e294e1193e Add reddit support 2022-10-31 15:10:27 +01:00
evazion
c76e0bd4c1 gelbooru: fix normalization of old image URLs. 2022-10-30 17:26:43 -05:00
nonamethanks
72528bdcb1 Twitter: rewrite tests 
Also add partial support for profile banner images.
 2022-10-17 18:53:11 +02:00
evazion
115521906c tumblr: fix failure to upload new Tumblr URLs containing the post title. 
Fix failure to upload Tumblr URLs of this form:

* https://www.tumblr.com/munespice/683613396085719040/saur-family
 2022-10-13 21:11:07 -05:00
evazion
268ec9118a tumblr: fix failure to upload certain video posts. 
Fix failure to upload Tumblr video posts that contained a video URL of this form:

* https://va.media.tumblr.com/tumblr_rjoh0hR8Xe1teimlz_720.mp4
 2022-10-13 21:10:29 -05:00
evazion
a07234121d tumblr: fixup for parsing www.tumblr.com/name URLs. 2022-10-13 00:26:16 -05:00
evazion
2e7b3cd80b tumblr: normalize https://www.tumblr.com/name artist URLs. 2022-10-12 23:55:17 -05:00
evazion
eb8f98e4a6 artists: normalize foriio.com artist URLs. 
Normalize `https://fori.io/comori22` to `https://www.foriio.com/comori22` in artist entries.
 2022-10-12 23:46:50 -05:00
evazion
f05268df7f sources: add Gelbooru support. 
Add support for uploading posts from Gelbooru. Note that the translated
tags will include both the Gelbooru tags and the tags from the Gelbooru
post's source. The commentary and artist information will also be taken
from the Gelbooru post's source. The source of the Danbooru post however
will be left as the Gelbooru post itself, not as the Gelbooru post's source.
 2022-10-11 00:06:45 -05:00
nonamethanks
d51cc17eaf Nicoseiga: rewrite tests and fix several bugs 
* Fixed a bug where manga posts with a single tag would raise an error
* Fixed a bug where dic.nicovideo.jp/oekaki posts weren't uploadable due
  to SSL issues
* Added support for more manga corner cases
 2022-09-29 14:37:46 +02:00
nonamethanks
5051c6649d Tumblr: parse new dashboard links 2022-09-28 17:00:08 +02:00
evazion
abf493794f twitter: fix misparsing of https://twitter.com/i/status/:id urls. 
Fix URLs like `https://twitter.com/i/status/943446161586733056` parsing
the username as `i`. This led to the new artist page recommending the
tag name `i` when creating an artist for a source like this.

Also fix these URLs not being normalized to `https://twitter.com/:username/status/:id` after upload.
 2022-09-15 19:57:12 -05:00
evazion
d2147eca80 tumblr: fix exception when fetching data for video urls. 
Fix an exception when trying to fetch source data for URLs like
https://va.media.tumblr.com/tumblr_pgohk0TjhS1u7mrsl.mp4.

For these URLs it's not possible to use the trick where we try to open
the URL as a HTML page and scrape the post id from the HTML. Instead we
get the raw video if we try to to this.
 2022-09-05 16:15:47 -05:00
evazion
f55951ab58 tumblr: fix exception when parsing mangled image urls. 
Fix a nil exception when trying to parse invalid URLs like `https://25.media.tumblr.com/91719d337b218681abc48cdc24e`.
 2022-09-05 16:15:46 -05:00
evazion
2b76a4c5ba tumblr: fix exception when parsing subdomainless Tumblr URLs. 
Fix exception when a post has a Tumblr source without a subdomain, such
as `https://tumblr.com`.
 2022-08-30 01:52:55 -05:00
evazion
4d009568fd Fix #5165: add support for weibo share urls 2022-08-26 01:12:23 -05:00
evazion
600bdc9ae6 pixiv: drop support for https://tc-pximg01.techorus-cdn.com urls. 
This was an obsolete URL format briefly used by Pixiv around 2019-2020.
There were only ~80 posts with sources using this format. They have been
manually fixed.
 2022-08-24 15:54:10 -05:00
evazion
bf3ee9cfb8 Fix #5238: Trying to upload a pixiv direct image url that got trumped by a revision redirects to the new post if it's uploaded. 
Bug: When uploading a direct Pixiv image URL, we ignored it in favor of the
image URL returned by the Pixiv API. This meant if you tried to upload the
original version of a revised image, we would get the revised version instead.

Fix: When given a direct Pixiv image URL, use it as-is if it's a full
image URL. If it's a sample image URL, ignore it in favor of the full image
URL as returned by the API, unless the post is deleted and the API data
is unavailable.
 2022-08-24 15:40:04 -05:00
evazion
f46134e87f Fix #5234: Weibo URLs get normalized incorrectly in some cases. 2022-08-24 14:47:00 -05:00
evazion
228850b749 newgrounds: support parsing video urls. 
Fixes URLS like `https://www.newgrounds.com/portal/view/830293` being treated as bad_source.
 2022-08-23 13:39:32 -05:00
evazion
9c2d362e93 tumblr: fix misparsing of image urls. 
Fix URLs like https://yogurtmedia.tumblr.com/post/45732863347 being
misparsed as image urls.
 2022-08-20 21:20:46 -05:00
evazion
9cab67c0ac artstation: fix parsing of reserved usernames. 2022-07-06 16:00:54 -05:00
nonamethanks
e7584c7e0a Nicoseiga: normalize oekaki links 2022-06-04 22:57:54 +02:00
evazion
6b54415c47 Merge pull request #5170 from nonamethanks/fix-fc2-bad-source 
Fc2: don't mark valid blog page sources as bad_source
 2022-05-16 15:12:07 -05:00
nonamethanks
dcbb2216aa Fc2: don't mark valid blog page sources as bad_source 2022-05-15 18:46:50 +02:00
evazion
c07b099bf8 Fix #5152: Nicovideo video urls getting bad_source. 2022-05-03 03:59:15 -05:00
evazion
2d9bba4abb posts: automatically add the bad_link and bad_source tags. 
Automatically add the bad_link tag when the source is an image url from
a known site, but it can't be converted to a page url (for example, a
Twitter or Tumblr direct image link).

Automatically add the bad_source tag when the source is from a known
site, but it's not an image or page url (for example, a Twitter or Pixiv
profile url)
 2022-05-01 21:01:36 -05:00
evazion
23b8350320 sources: add image_url?, page_url?, and profile_url? methods. 
Add methods to Source::URL for determining whether a URL is an image
URL, a page URL, or a profile URL.

Also add more source URL tests and fix various URL parsing bugs.
 2022-05-01 21:01:36 -05:00
nonamethanks
8edd5dd810 Add furaffinity support 2022-04-27 03:47:59 +02:00
evazion
90182148aa Merge pull request #5137 from nonamethanks/foundation-videos 
Foundation: fix some video posts not being extracted
 2022-04-22 01:50:26 -05:00
evazion
57a92ad336 Fix #5072: Fandom source normalization is wrong 2022-04-22 01:27:17 -05:00