evazion
23b8350320
sources: add image_url?, page_url?, and profile_url? methods.
...
Add methods to Source::URL for determining whether a URL is an image
URL, a page URL, or a profile URL.
Also add more source URL tests and fix various URL parsing bugs.
2022-05-01 21:01:36 -05:00
evazion
90182148aa
Merge pull request #5137 from nonamethanks/foundation-videos
...
Foundation: fix some video posts not being extracted
2022-04-22 01:50:26 -05:00
nonamethanks
e1b9166a56
Sources: do not use an empty else in case blocks
2022-04-22 03:53:18 +02:00
nonamethanks
e6cb255a7a
Foundation: fix some video posts not being extracted
...
Also adjusts SourceTestHelper to not autogenerate contexts, so that
tests can be launched individually.
2022-04-21 17:54:22 +02:00
evazion
3aa5cab2aa
sources: refactor normalize_for_source.
...
`normalize_for_source` was used to convert image URLs to page URLs when displaying sources
on the post show page. Move all the code for converting image URLs to page URLs from
`Sources::Strategies#normalize_for_source` to `Source::URL#page_url`.
Before we had to be very careful in source strategies not to make any network calls in
`normalize_for_source`, since it was used in the view for the post show page. Now all the
code for generating page URLs is isolated in Source::URL, which makes source strategies
simpler. It also makes it easier to check if a source is an image URL or page URL, and if
the image URL is convertible to a page URL, which will make autotagging bad_link or
bad_source feasible.
Finally, this fixes it to generate better page URLs in a handful of cases:
* https://www.artstation.com/artwork/qPVGP instead of https://anubis1982918.artstation.com/projects/qPVGP
* https://yande.re/post/show?md5=b4b1d11facd1700544554e4805d47bb6s instead of https://yande.re/post?tags=md5:b4b1d11facd1700544554e4805d47bb6
* http://gallery.minitokyo.net/view/365677 instead of http://gallery.minitokyo.net/download/365677
* https://valkyriecrusade.fandom.com/wiki/File:Crimson_Hatsune_H.png instead of https://valkyriecrusade.wikia.com/wiki/File:Crimson_Hatsune_H.png
* https://rule34.paheal.net/post/view/852405 instead of https://rule34.paheal.net/post/list/md5:854806addcd3b1246424e7cea49afe31/1
2022-03-23 01:34:04 -05:00
evazion
9343f7c912
Source::URL: add profile_url method.
...
Add a method for converting a source URL into a profile URL. This will
be used for normalizing profile URLs in artist entries.
Also add the ability to parse a few more profile URL formats.
2022-03-13 03:54:17 -05:00
nonamethanks
ebd3670076
sources: remove redundant comments
...
These comments are already present under the parse blocks, so the huge
walls of text before the code are not needed anymore.
2022-03-08 16:56:00 +01:00
nonamethanks
d195d30587
Foundation: fix normalization error
...
Urls like https://foundation.app/@yohan1754/fso/3 would get normalized
like https://foundation.app/@foundation/foundation/3 , which was wrong
because it would point to a completely different collection
2022-03-07 06:52:23 +01:00
evazion
926a8fa81f
Danbooru::URL: add #basename, #filename, and #file_ext utility methods.
...
Add `#basename`, `#filename`, and `#file_ext` utility methods to
Danbooru::URL and change a few places to use them. Simplifies parsing
filenames in source URLs in various places.
2022-02-27 02:27:21 -06:00
evazion
ffe52f5ead
sources: factor out Source::URL::Foundation.
...
Add support for a couple more URL types:
* https://foundation.app/@asuka111art/dinner-with-cats-82426
* https://f8n-production-collection-assets.imgix.net/0x3B3ee1931Dc30C1957379FAc9aba94D1C48a5405/128711/QmcBfbeCMSxqYB3L1owPAxFencFx3jLzCPFx6xUBxgSCkH/nft.png
Also include these URLs in the list of profile URLs:
* https://foundation.app/0x7E2ef75C0C09b2fc6BCd1C68B6D409720CcD58d2 (for https://foundation.app/@mochiiimo )
These URLs should be stable even if the user changes their name.
2022-02-23 23:49:31 -06:00
