Fix regression in #4475. Fetch the commentary as html instead of
plaintext so that we don't lose links or other formatting.
Also fix it so that /jump.php redirect links are replaced with the
actual url.
* Move the source normalization logic out of the post model
and into individual sources' strategies.
* Rewrite normalization tests to be handled into each source's test,
and expand them significantly. Previously we were only testing
a very small subset of domains and variants.
* Fix up normalization for several sites.
* Normalize fav.me urls into normal deviantart urls.
that's the latest commit made to deviantart files before switching from
the developer API to the Javascript backend from the new "Eclipse"
frontend.
This is necessary because it's basically impossible to download posts
now with the JS backend without being logged in, i.e. having the cookies
from a logged in user, which can't be used for very long even if
exporting them from a browser. You would have to save the cookies
deviantart sends you back via the "Set-Cookie" header in a database
somewhere in addition to the other added complexity.
also
* (temporarily) replace HttpartyCache with HTTParty as it's long been
removed
* fix one case of "last argument as keyword parameter"
* change repository url (5d1a1cc87e)
* remove self-explanatory comment
Add test for #4243. Also fix warning from bootsnap:
iseq.rb:18: warning: nested repeat operator '+' and '?' was replaced with '*' in regular expression: /(?<!\A)生誕祭(?:\d+)?\z/
The twitter gem had several problems:
* It's been unmaintained for over a year.
* It pulled in a lot of dependencies, many of which were outdated. In
particular, it locked the `http` gem to version 3.3, preventing us
from upgrading to 4.2.
* It raised exceptions on normal error conditions, like for deleted
tweets or suspended users, which we really don't want.
* We had to wrap it to provide caching.
Changes:
* Fixes#4226 (Exception when creating new artists entries for suspended
Twitter accounts)
* Drop support for scraping images from summary cards. Summary cards
are the previews you get when you link to a website in a tweet. These
preview images aren't always the best image.
Fix the moebooru strategy to fallback to returning the image url if we
can't find the preview url. Fixes iqdb lookups failing in some cases
because the strategy didn't return a valid url for preview_url.
* Tighten up illust id parsing to avoid misparsing ids from
non-illust urls (sketch urls and novel urls).
* Move id parsing tests from post_test.rb to sources/pixiv_test.rb.
* Drop support for touch.pixiv.net urls. These urls are no longer used
by Pixiv and aren't present as the source of any posts on Danbooru.
Bug: Uploading bad pixiv id images failed because the pixiv strategy
raised a BadIDError exception when the upload service checked for the
ugoira frame data.
* Normalize spaces to underscores when saving other names. Preserve case
since case can be significant.
* Fix WikiPage#other_names_include to search case-insensitively (note:
this prevents using the index).
* Fix sources to return the raw tags in `#tags` and the normalized tags
in `#normalized_tags`. The normalized tags are the tags that will be
matched against other names.
