Commit Graph

2833 Commits

Author SHA1 Message Date
evazion
7a1efc2744 Fix #4522: Sidebar doesn't show most searched tags at certain times of day.
Revert back to previous workaround of fetching previous day if current
day returns no result. A terrible hack, really we should convert dates
to Reportbooru's timezone, but that has other complications.
2020-06-19 14:00:50 -05:00
evazion
67a52dbc2d tumblr: support new va.media.tumblr.com urls. 2020-06-19 13:53:35 -05:00
nonamethanks
0a396c8b95 Revert "Pixiv: don't blacklist digital tools"
This reverts commit e83d07ea7b.

It was worth a try, but unfortunately it seems that once
someone sets tools in a Pixiv upload, they become defaults and
are applied to all of their subsequent uploads, so we get some
posts with two or three different digital tags.
2020-06-19 08:08:46 +02:00
nonamethanks
8a06d3a744 Zerochan: Normalize png links 2020-06-18 08:07:45 +02:00
evazion
459f67c431 iqdb: fix 599 timeout errors.
Increase timeout to 30 seconds when uploading files to IQDB. Previously
we used the default timeout of 3 seconds, which could cause 599 timeout
errors sometimes if the upload took too long.
2020-06-18 00:57:51 -05:00
evazion
f790a1aeed http: increase default timeout to 10 seconds.
Three seconds was little tight and might have caused timeouts in source
strategies sometimes.
2020-06-18 00:56:42 -05:00
nonamethanks
7a41ee9c34 Add NewGrounds support 2020-06-18 03:22:30 +02:00
evazion
d5a7eef53d uploads: fix remote file size not appearing. 2020-06-17 04:13:16 -05:00
evazion
b8b5c8d6a0 iqdb: fix error with file uploads.
Fix "cannot determine size of body" errors on upload page. Caused by
exception during IQDB lookup. We were posting the form data wrong, we
need to wrap the file with HTTP::FormData::File and pass it through the
`form` parameter.
2020-06-17 02:28:16 -05:00
BrokenEagle
158a4aa916 Fix Pixiv user profile URL to use the latest format
This will only affect new artist and commentary records going forward.
2020-06-17 07:07:33 +00:00
evazion
91beb288b6 reportbooru: cache post views endpoint.
Cache most viewed posts endpoint for /explore/posts/viewed page.
2020-06-17 00:57:55 -05:00
evazion
b551e3634f Fix misc rubocop warnings. 2020-06-16 21:36:15 -05:00
evazion
101e8a13b8 /posts: fix blank tag sidebars.
Fix the sidebar on the /posts index page sometimes being blank. This
could happen when either the related tag calculation was too slow and
timed out, or when Reporbooru was unavailable and we couldn't fetch the
list of popular tags.

In the tag list would otherwise be blank, we fall back to frequent tags
(the most common tags on the current page of results).

Also change it so that if Reportbooru is unconfigured, we fail
gracefully by returning blank results instead of failing with an
exception. This is so we can still view the popular searches and missed
searches pages during testing (even though they'll be blank).
2020-06-16 12:58:41 -05:00
evazion
1aa0f65187 sources: fix rubocop warnings. 2020-06-16 00:10:37 -05:00
evazion
7868e5045e nicoseiga: fix regression with http redirects.
3cdf67920 changed it so that Danbooru::Http follows redirects by
default. This broke some things in the Nico Seiga strategy, so disable
following redirects in the Nico Seiga API client for now.

Also change it so that Danbooru::Http follows redirects after a POST
request (by setting `strict: false`). Nico Seiga needs this because it
sends a redirect after we POST the login form.
2020-06-15 22:13:28 -05:00
evazion
962e60f4f1 Merge pull request #4496 from nonamethanks/refactor_nicoseiga
Refactor nicoseiga
2020-06-15 21:05:10 -05:00
evazion
19727ab5c4 Merge pull request #4505 from nonamethanks/pixiv_digital_tags
Pixiv: don't blacklist digital tools anymore
2020-06-15 20:56:56 -05:00
nonamethanks
5b186f3072 Support for new nicoseiga cdn domain 2020-06-15 04:01:34 +02:00
nonamethanks
6fc4d3ec44 Nicoseiga: Add support for drm-served manga 2020-06-15 03:37:51 +02:00
nonamethanks
260bc997f6 NicoSeiga: Add preview urls 2020-06-15 03:37:51 +02:00
nonamethanks
9f0e85e1b5 Refactor nicoseiga strategy
* Get rid of mechanize, fully switch to Danbooru::Http
* Switch to mobile api, improving speed
* Merge main and manga clients
* Add full support for manga pages
* Add support for anonymous and r-15 images
* Don't fail when attempting to upload oekaki direct links
* Various misc fixes
2020-06-15 03:37:51 +02:00
evazion
3cdf679202 http: automatically follow redirects.
* Automatically follow redirects (up to 5, return a synthetic 598 error after that).
* Add `put` and `timeout` methods.
* Add tests.
2020-06-14 01:39:55 -05:00
evazion
cd501fe27b iqdb: switch to Danbooru::Http. 2020-06-14 01:06:51 -05:00
evazion
a4df18e650 Refactor Reportbooru API clients.
* Combine MissedSearchService, PostViewCountService, and
  PopularSearchService into single ReportbooruService class.
* Use Danbooru::Http for these services instead of HTTParty.
2020-06-14 00:32:42 -05:00
evazion
1846133cd6 post view count service: switch to Danbooru::Http. 2020-06-13 21:50:06 -05:00
nonamethanks
e83d07ea7b Pixiv: don't blacklist digital tools anymore 2020-06-12 04:15:20 +02:00
evazion
8a2ae91ff2 tests: skip video file tests if ffmpeg isn't installed. 2020-06-10 18:07:54 -05:00
evazion
15799f8af7 Fix #4260: Unable to replace cdn.donmai.us images?
Bug: Replacing posts hosted on cdn.donmai.us didn't work.

Cause: Original files on cdn.donmai.us are hosted under /var/www/danbooru/original/, but replacements
were trying to store them directly under /var/www/danbooru, which failed with a permission error.
We were trying to store them in the wrong directory because we didn't respect the `original_subdir`
option when generating file paths.
2020-06-09 15:57:43 -05:00
evazion
d002701bc1 Merge pull request #4494 from nonamethanks/fix_deviantart_api_downloads
Deviantart: fix api downloads
2020-06-09 01:37:03 -05:00
evazion
484eacfd3b config: remove is_unlimited_metatag? config option. 2020-06-02 18:25:42 -05:00
evazion
9997db44d8 config: remove banned_ip_for_download? option.
* Remove `banned_ip_for_download?` config option. This isn't something that usually needs
  to be configured.
* Replace the `ipaddress` gem with `ipaddress_2`. The `ipaddress` gem has several methods
  we need (`link_local?`, etc) that are only available in master because the gem hasn't had
  an official release in several years. `ipaddress_2` is a fork that is more actively
  maintained.
2020-06-02 15:32:19 -05:00
nonamethanks
25b801619f Deviantart: fix api downloads 2020-05-31 07:01:43 +02:00
evazion
855e31ac90 nijie: fetch commentary as html instead of plaintext.
Fix regression in #4475. Fetch the commentary as html instead of
plaintext so that we don't lose links or other formatting.

Also fix it so that /jump.php redirect links are replaced with the
actual url.
2020-05-29 15:36:21 -05:00
evazion
206ac7dd9a dtext#from_html: convert basic links to <url> syntax.
Convert

    <a href="https://www.example.com">https://www.example.com</a>

to

    <https://www.example.com>

instead of

    "https://www.example.com":[https://www.example.com]
2020-05-29 15:36:21 -05:00
evazion
88d9fc4e5e sources: simplify artist finder url normalization.
Get rid of `normalized_for_artist_finder?` and `normalizable_for_artist_finder?`.
This was legacy bullshit that was originally designed to avoid API calls
when saving artist entries containing old Pixiv direct image urls that
had already been normalized, or that couldn't be normalized because they
were bad id.

Nowadays we store profile urls in artist entries instead of direct image
urls, so we don't normally need to do any API calls to normalize the
profile url. Strategies should take care to avoid triggering API calls
inside `profile_url` when possible.
2020-05-29 15:35:15 -05:00
nonamethanks
d339947647 Weibo: add source normalization 2020-05-28 01:05:11 +02:00
evazion
feeea6602c Merge pull request #4488 from nonamethanks/add_weibo_support
Add Weibo support
2020-05-27 16:53:14 -05:00
evazion
2c60a51f64 Merge pull request #4475 from nonamethanks/refactor_source_normalizing
Refactor source normalization
2020-05-27 16:52:17 -05:00
evazion
241894428a Merge pull request #4480 from BrokenEagle/fix-artstation
Fixes issues with Artstation source strategy
2020-05-27 15:37:23 -05:00
evazion
7c5510a650 media_file/flash: add license information (#4484). 2020-05-27 15:11:46 -05:00
evazion
f94c52478c media file: memoize expensive methods in subclasses. 2020-05-27 14:31:39 -05:00
nonamethanks
5c7307a1c9 Add Weibo support 2020-05-27 11:30:05 +02:00
lllusion3469
5c2ecee60f uploads: memoize dimensions of flash files
flash files can be quite big (the biggest on danbooru.donmai.us being
68.6MB atm). Reading it and applying complex transformations twice seems
unnecessary.
2020-05-25 18:58:48 +02:00
lllusion3469
b2814c1125 uploads: fix getting dimensions of flash files
MediaFile#dimensions is called twice - in #width and in #height but
it only works on the first call because the file is read to the end and
consumed the first time so when #read is called the second time it only
returns the empty string
2020-05-25 18:51:36 +02:00
evazion
c300b344de Merge pull request #4481 from BrokenEagle/fix-iqdb-image-url
Adds suport for using the image_url parameter with IQDB
2020-05-25 02:20:55 -05:00
evazion
e22e403bac Merge pull request #4478 from nonamethanks/fix_nijie_truncated_commentary
Nijie: fetch full commentary rather than truncated preview
2020-05-25 02:13:24 -05:00
evazion
20f8a26709 tests: fix rails 2.7 keyword parameter deprecation warnings. 2020-05-25 01:48:46 -05:00
evazion
3e2949d157 uploads: fix exception when uploading gifs.
Older versions of libvips don't support get("n-pages"). This is known to
fail in libvips-8.4 and known to work in libvips-8.8.
2020-05-24 12:07:17 -05:00
evazion
dc4a3bcf38 uploads: fix incorrect sample image sizing.
ref: https://danbooru.donmai.us/forum_topics/16935.

Bug: sample images were being generated to be at most 850px width *and*
850px tall. They're supposed to be at most 850px wide with unlimited height.
2020-05-23 23:29:59 -05:00
BrokenEagle
b9b1177832 Adds suport for using the image_url parameter with IQDB
This was only halfways supported, as the download module does not
have an image_url function. So for this, it just uses the url function,
which is just the original URL passed into the download function.

Additionally, it adds support to grab the largest available image,
which it does by using the file_url function of the downloads module.

- Fixes image_url parameter
- Adds file_url parameter
2020-05-24 00:58:46 +00:00