Fix the sidebar on the /posts index page sometimes being blank. This
could happen when either the related tag calculation was too slow and
timed out, or when Reporbooru was unavailable and we couldn't fetch the
list of popular tags.
In the tag list would otherwise be blank, we fall back to frequent tags
(the most common tags on the current page of results).
Also change it so that if Reportbooru is unconfigured, we fail
gracefully by returning blank results instead of failing with an
exception. This is so we can still view the popular searches and missed
searches pages during testing (even though they'll be blank).
3cdf67920 changed it so that Danbooru::Http follows redirects by
default. This broke some things in the Nico Seiga strategy, so disable
following redirects in the Nico Seiga API client for now.
Also change it so that Danbooru::Http follows redirects after a POST
request (by setting `strict: false`). Nico Seiga needs this because it
sends a redirect after we POST the login form.
* Get rid of mechanize, fully switch to Danbooru::Http
* Switch to mobile api, improving speed
* Merge main and manga clients
* Add full support for manga pages
* Add support for anonymous and r-15 images
* Don't fail when attempting to upload oekaki direct links
* Various misc fixes
* Combine MissedSearchService, PostViewCountService, and
PopularSearchService into single ReportbooruService class.
* Use Danbooru::Http for these services instead of HTTParty.
Bug: Replacing posts hosted on cdn.donmai.us didn't work.
Cause: Original files on cdn.donmai.us are hosted under /var/www/danbooru/original/, but replacements
were trying to store them directly under /var/www/danbooru, which failed with a permission error.
We were trying to store them in the wrong directory because we didn't respect the `original_subdir`
option when generating file paths.
* Remove `banned_ip_for_download?` config option. This isn't something that usually needs
to be configured.
* Replace the `ipaddress` gem with `ipaddress_2`. The `ipaddress` gem has several methods
we need (`link_local?`, etc) that are only available in master because the gem hasn't had
an official release in several years. `ipaddress_2` is a fork that is more actively
maintained.
Fix regression in #4475. Fetch the commentary as html instead of
plaintext so that we don't lose links or other formatting.
Also fix it so that /jump.php redirect links are replaced with the
actual url.
Get rid of `normalized_for_artist_finder?` and `normalizable_for_artist_finder?`.
This was legacy bullshit that was originally designed to avoid API calls
when saving artist entries containing old Pixiv direct image urls that
had already been normalized, or that couldn't be normalized because they
were bad id.
Nowadays we store profile urls in artist entries instead of direct image
urls, so we don't normally need to do any API calls to normalize the
profile url. Strategies should take care to avoid triggering API calls
inside `profile_url` when possible.
flash files can be quite big (the biggest on danbooru.donmai.us being
68.6MB atm). Reading it and applying complex transformations twice seems
unnecessary.
MediaFile#dimensions is called twice - in #width and in #height but
it only works on the first call because the file is read to the end and
consumed the first time so when #read is called the second time it only
returns the empty string
ref: https://danbooru.donmai.us/forum_topics/16935.
Bug: sample images were being generated to be at most 850px width *and*
850px tall. They're supposed to be at most 850px wide with unlimited height.
This was only halfways supported, as the download module does not
have an image_url function. So for this, it just uses the url function,
which is just the original URL passed into the download function.
Additionally, it adds support to grab the largest available image,
which it does by using the file_url function of the downloads module.
- Fixes image_url parameter
- Adds file_url parameter
The reason that the download was failing was not because the 4k size
didn't exist, but because the Artstation had no way to handle image
cover URLs. This caused it to pass nil to the download function.
Additionally, there was no way to get the preview URL size, i.e. the
smallest available image for an Arstation image URL.
- Adds support for cover URLs
- Adds support for preview URL size
* Move the source normalization logic out of the post model
and into individual sources' strategies.
* Rewrite normalization tests to be handled into each source's test,
and expand them significantly. Previously we were only testing
a very small subset of domains and variants.
* Fix up normalization for several sites.
* Normalize fav.me urls into normal deviantart urls.
* Move image thumbnail generation code to MediaFile::Image.
* Move video thumbnail generation code to MediaFile::Video.
* Move ugoira->webm conversion code to MediaFile::Ugoira.
This separates thumbnail generation from the upload process so that it's
possible to generate thumbnails outside of uploads.
Fixes bug described in d3e4ac7c17 (commitcomment-39049351)
When dealing with searches, there are several variables we have to keep
in mind:
* Whether tag aliases should be applied.
* Whether search terms should be sorted.
* Whether the rating:s and -status:deleted metatags should be added by
safe mode and the hide deleted posts setting.
Which of these things we need to do depends on the context:
* We want to apply aliases when actually doing the search, calculating
the count, looking up the wiki excerpt, recording missed/popular
searches in Reportbooru, and calculating related tags for the sidebar,
but not when displaying the raw search as typed by the user (for
example, in the page title or in the tag search box).
* We want to sort the search when calculating cache keys for fast_count
or related tags, and when recording missed/popular searches, but not
in the page title or when displaying the raw search.
* We want to add rating:s and -status:deleted when performing the
search, calculating the count, or recording missed/popular searches,
but not when calculating related tags for the sidebar, or when
displaying the page title or raw search.
Here we introduce normalized_query and try to use it in contexts where
query normalization is necessary. When to use the normalized query
versus the raw unnormalized query is still subtle and prone to error.
This was a hack to deal with the Cloudflare check sometimes being slow
or timing out during tests. The call to https://api.cloudflare.com/client/v4/ips
could hang if there were IPv6 connectivity problems. If this happens, make
sure that IPv6 is configured properly and that `curl -v --http1.1 -6 https://api.cloudflare.com/client/v4/ips`
works.
* Add a three second connection timeout to all http requests. By default
http.rb doesn't have any timeouts, so it can hang forever trying to
connect if there are any network issues.
* Return a fake 522 error in the event of a timeout so that callers
don't have to deal with TimeoutError exceptions, instead they can treat
timeouts as normal 5xx errors (which most callers already handle).
