Rework sitemaps to provide more coverage of the site. We want every
important page on the site - including every post, tag, and wiki page -
to be indexed by Google. We do this by generating sitemaps and sitemap
indexes that contain links to every important page on the site.
Bug: if you created an artist with the name of an existing general tag,
then the gentag would be changed to an artist tag, no matter how big the
gentag was.
Now we only allow creating artist entries for non-artist tags if the tag
is empty.
Ref: https://danbooru.donmai.us/forum_topics/17095
ArtStation is apparently returning images with smaller filesizes for
/4k/ images now. It doesn't look like we can get the old images, so just
update the tests.
Bug: when a search timed out we got the generic failbooru page instead
of the search timeout error page.
Cause: when rendering the <link rel="next"> / <link rel="prev"> tags in
the header, we may need to evaluate the search to determine the next or
previous page, but if the searches times out then this fails, which
caused Rails to throw a ActionView::Template::Error because an exception
was thrown while rendering the template.
Likewise, rendering the attributes for the <body> tag could fail with an
ActionView::Template::Error because the call to `current_item.present?`
forced evaluation of the search.
Bug: The frontpage failed due to a SSL error. We couldn't fetch the
popular tag list from Reportbooru because Reportbooru's SSL certificate
had expired and HTTP.rb raised an SSLError exception that we didn't
catch.
Fix: Convert the SSLError to a 5xx HTTP error to prevent SSL exceptions
from leaking through HTTP.rb.
Remove sending dmail notifications to uploaders when an upload is
disapproved. These messages are usually confusing and frustrating to
uploaders. They don't know who is sending them and they usually feel
insulted when they get negative messages from anonymous users.
* Fix incorrect canonical tags. Before we were using
`<meta name="canonical" content="...">`. This is wrong, it should have been
`<link rel="canonical" href="...">`.
* Add a default canonical tag on all pages. Fixes Google treating the
same content on different subdomains (safebooru, shima, kagamihara, etc)
as duplicate content. Also fixes Google sometimes treating similar but
distinct content on the same domain as duplicate content.
Fix the wiki excerpt not appearing when searching for a tag that doesn't
exist in the tag list. This could happen if someone created a wiki for a
tag that has never been used on a post.
* Factor out the Cloudflare Polish bypass code to a standalone feature.
* Add `http_downloader` method to the base source strategy. This is a
HTTP client that should be used for downloading images or making
requests to images. This client ensures that referrer spoofing and
Cloudflare bypassing are performed.
This fixes a bug with the upload page reporting the polished filesize
instead of the original filesize when uploading ArtStation images.
Factor out referrer spoofing so that it can be used outside of downloading
files. We also need to spoof the referrer when determining the remote
filesize of images on the uploads page.
Bug: the uploads page showed a remote size of 146 bytes for Pixiv uploads.
Cause: we didn't spoof the Referer header when making the HEAD request
for the image, causing Pixiv to return a 403 error.
Also fix the case where the Content-Length header is absent.
Add a `respond_to_search` test helper for concisely testing that a
controller's index action correctly responds to a search. Usage:
# Tests that `/tags.json?search[name]=touhou` returns the `touhou` tag.
setup { @touhou = create(:tag, name: "touhou") }
should respond_to_search(name: "touhou").with { @touhou }
These exceptions are no longer thrown now that we've switched from
HTTParty to http.rb. Swallowing unexpected exceptions during testing was
a bad practice anyway.
Replace the mocked services in scripts/mocked_services with Rails-level
mocked services.
The scripts in scripts/mocked_services were a set of stub Sinatra
servers used to mock the Reportbooru, Recommender, and IQDBs services
during development. They return fake data so you can test pages that use
these services.
Implementing these services in Rails makes it easier to run them. It
also lets us drop a dependency on Sinatra and drop a use of HTTParty.
To use these services, set the following configuration in danbooru_local_config.rb
or .env.local:
* reportbooru_server: http://localhost:3000/mock/reportbooru
* recommender_server: http://localhost:3000/mock/recommender
* iqdbs_server: http://localhost:3000/mock/iqdb
where `http://localhost:300` is the url for your local Danbooru server
(may need to be changed depending on your configuration).
The Nijie login process works like this:
* First we submit our `email` and `password` to `https://nijie.info/login_int.php`.
* Then we save the NIJIEIEID session cookie from the response.
* We optionally retry if login failed. Nijie returns 429 errors with a
`Retry-After: 5` header if we send too many login requests. This can
happen during parallel testing.
* We cache the login cookies for only 1 hour so we don't have to worry
about them becoming invalid if we cache them too long.
Cookies and retrying errors on failure are handled transparently by Danbooru::Http.
Allow cookies to be saved and sent back when making several requests in
a row. Usage:
http = Danbooru::Http.use(:session)
# saves the foo=42 cookie sent by the response.
http.get("https://httpbin.org/cookies/set/foo/42")
# sends back the foo=42 cookie from the previous request.
http.get("https://httpbin.org/cookies")
evazion
evazion
BrokenEagle