7ed8f95a8e
Introduce a Source::URL class for parsing URLs from source sites. Refactor the Twitter source strategy to use it. This is the first step towards factoring all the URL parsing logic out of source strategies and moving it to subclasses of Source::URL. Each site will have a subclass of Source::URL dedicated to parsing URLs from that site. Source strategies will use these classes to extract information from URLs. This is to simplify source strategies. Most sites have many different URL formats we have to parse or rewrite, and handling all these different cases tends to make source strategies very complex. Isolating the URL parsing logic from the site scraping logic should make source strategies easier to maintain.
67 lines
2.0 KiB
Ruby
67 lines
2.0 KiB
Ruby
# frozen_string_literal: true
|
|
|
|
# A Source::URL is a URL from a source site, such as Twitter, Pixiv, etc. Each site has a
|
|
# subclass responsible for parsing and extracting information from URLs for that site.
|
|
#
|
|
# To add a new site, create a subclass of Source::URL and implement `#match?` to define
|
|
# which URLs belong to the site, and `#parse` to parse and extract information from the URL.
|
|
#
|
|
# Source::URL is a subclass of Danbooru::URL, so it inherits some common utility methods
|
|
# from there.
|
|
#
|
|
# @example
|
|
# url = Source::URL.parse("https://twitter.com/yasunavert/status/1496123903290314755")
|
|
# url.site_name # => "Twitter"
|
|
# url.status_id # => "1496123903290314755"
|
|
# url.twitter_username # => "yasunavert"
|
|
#
|
|
module Source
|
|
class URL < Danbooru::URL
|
|
SUBCLASSES = [
|
|
Source::URL::Twitter,
|
|
]
|
|
|
|
# Parse a URL into a subclass of Source::URL, or raise an exception if the URL is not a valid HTTP or HTTPS URL.
|
|
#
|
|
# @param url [String, Danbooru::URL]
|
|
# @return [Source::URL]
|
|
def self.parse!(url)
|
|
url = Danbooru::URL.new(url)
|
|
subclass = SUBCLASSES.find { |c| c.match?(url) } || Source::URL
|
|
subclass.new(url)
|
|
end
|
|
|
|
# Parse a string into a URL, or return nil if the string is not a valid HTTP or HTTPS URL.
|
|
#
|
|
# @param url [String, Danbooru::URL]
|
|
# @return [Danbooru::URL]
|
|
def self.parse(url)
|
|
parse!(url)
|
|
rescue Error
|
|
nil
|
|
end
|
|
|
|
# Subclasses should implement this to return true for URLs that should be handled by the subclass.
|
|
#
|
|
# @param url [Danbooru::URL] The source URL.
|
|
def self.match?(url)
|
|
raise NotImplementedError
|
|
end
|
|
|
|
# @return [String, nil] The name of the site this URL belongs to, or possibly nil if unknown.
|
|
def site_name
|
|
self.class.name.demodulize
|
|
end
|
|
|
|
protected def initialize(...)
|
|
super(...)
|
|
parse
|
|
end
|
|
|
|
# Subclasses should implement this to parse and extract any useful information from
|
|
# the URL. This is called when the URL is initialized.
|
|
protected def parse
|
|
end
|
|
end
|
|
end
|