Commit Graph
977 Commits
Author SHA1 Message Date
evazion f5c9616f62 tests: fix broken tests. 2018-09-24 17:34:08 -05:00
evazion d9063a9f2a nijie: support preview urls (#3919). 2018-09-24 17:08:37 -05:00
evazion 5525bbe1ca nijie: normalize all thumbnail urls (#3919). 2018-09-23 20:08:14 -05:00
evazion d294514dc0 nijie: don't crash on invalid urls or deleted works (#3919). 2018-09-23 20:08:14 -05:00
evazion b6228505aa nijie: fix page_url method.
The id in a bare image url is the member id, not the illust id.
2018-09-23 20:08:13 -05:00
evazion 39f9e01b13 nicoseiga: fix canonical_url to use the image url. 2018-09-22 11:07:18 -05:00
evazion 09a8198979 /artists: add wildcard, regex search to url field (#3900)
Allow searching the URL field by regex or by wildcard.

If the query looks like `/twitter/` do a regex search, otherwise if it
looks like `http://www.twitter.com/*` do a wildcard search, otherwise if
it looks like an url do an artist finder search, lastly if it looks like
`twitter` do a `*twitter*` search.
2018-09-21 21:19:01 -05:00
evazion a4608daf38 /artists: add more search options for other names, group name.
Add these search params:

* /artists?search[<field>]=
* /artists?search[<field>_eq]=
* /artists?search[<field>_not_eq]=
* /artists?search[<field>_like]=
* /artists?search[<field>_not_like]=
* /artists?search[<field>_ilike]=
* /artists?search[<field>_not_ilike]=
* /artists?search[<field>_regex]=
* /artists?search[<field>_not_regex]=

where `<field>` can be `name`, `group_name`, or `other_names`.

Remove these search params:

* /artists?search[name_matches]=
* /artists?search[other_names_match]=
* /artists?search[group_name_matches]=

`/artists?search[<field>_like]=` effectively does the same thing that
these searches did.
2018-09-21 20:55:14 -05:00
evazion f917b83d6f /artists: drop deprecated search syntax, add regex search for names.
Drop support for the following pseudo-metatags in the Name field in the
artists search form:

* name:<name>
* other:<other name>
* group:<group name>
* status:banned
* status:active
* http://www.example.com

Instead, make the Name field do a wildcard search against the artist
name, group name, or other names. If the query looks like `/regex/`,
then do a regex search against any of these names.

/artists?search[name] now does a literal exact match and
/artists?search{any_name_matches] does the above wildcard/regex search.
2018-09-21 20:51:53 -05:00
evazion 237ab9b782 dmail filters: fix filters being case sensitive. 2018-09-20 19:51:59 -05:00
evazion 03abbd0683 Fix #2894: Use [[:space:]] instead of \s in regexes. 2018-09-20 19:24:38 -05:00
evazion 29cdaddd86 PostSetPresenters::Post#related_posts: clean up metatag parsing (#2894).
* Fix `#related_tags` to use `Tag.has_metatag?`.
* Fix Tag::SUBQUERY_METATAGS and Tag::METATAGS to be arrays instead of regexes.
2018-09-20 19:23:47 -05:00
evazion 6fe883c316 posts index: clean up limit:<n> / order:random parsing (#2894)
* Move the limit:<n> / order:random metatag parsing from the controller
  to the post set.

* Introduce `Tag.has_metatag?` and use it to parse these metatags
  instead of using a regex (#2894).
2018-09-20 19:23:47 -05:00
evazion 958a9f505b moebooru: rewrite sample urls + support bookmarklet on html page.
* Fixes #2942: Add Moebooru Rewrite for Sample Images.
* Addresses #3911: Improve Moebooru support.
2018-09-19 23:32:21 -05:00
evazion f4e08ef30d Downloads::File: fix SSRF inside is_cloudflare? (#2498).
Fixes the banned IP check not being applied when sending the HEAD
request for is_cloudflare?.

Also fixes the `#size` method not using the uncached url (which meant
the bookmarklet could report the wrong filesize on artstation uploads).
2018-09-19 20:11:53 -05:00
Albert Yi 9e1e73ec4c migrate blacklists during mass updates (#2251) 2018-09-19 16:25:06 -07:00
evazion 2f17082e73 Downloads::File: fix SSRF when following redirects (#2498).
Fixes the banned IP check not being applied when following redirects:

  http://danbooru.donmai.us/uploads/new?url=http://httpbin.org/redirect-to%3Furl=http://127.0.0.1/test.jpg
2018-09-18 16:00:03 -05:00
evazion 99221e4028 Downloads::File: fix SSRF attack when fetching remote size (#2498).
Fixes the banned IP check not being applied when fetching the remote
file size. This allowed one to trick Danbooru into sending HEAD requests
to private IPs:

  http://danbooru.donmai.us/uploads/new?url=http://127.0.0.1/test.jpg
2018-09-18 12:16:27 -05:00
evazion 9cdfbba6c2 Fix #3910: Corrupted images during upload.
Use a fresh tempfile for each download attempt instead of reusing the same
file (and having to rewind/truncate it after each failed attempt).
2018-09-18 10:01:44 -05:00
evazion d3c135ec72 Downloads::File#http_get_streaming: clean up retry logic.
Replace handrolled retry logic with retriable gem (already pulled in by another gem).
2018-09-18 09:44:15 -05:00
evazion 96e89cecfb tests: move twitter canonical url test. 2018-09-17 23:27:53 -05:00
Albert Yi aea2d77c3a Merge pull request #3907 from evazion/fix-3900
#3900: Allow to search for urls associated with artists using wildcards
2018-09-17 12:03:15 -07:00
evazion 4a99cb098f moebooru: use the image url as the canonical url. 2018-09-16 21:00:11 -05:00
evazion d9ce953752 Fix #3906: Moebooru strategy raises NotImplementedError. 2018-09-16 21:00:11 -05:00
evazion cae78fa8ee moebooru: move tests from unit/downloads to unit/sources. 2018-09-16 21:00:11 -05:00
evazion f135a7c064 twitter: normalize canonical urls.
Normalize http://mobile.twitter.com to http://twitter.com in canonical urls.
2018-09-16 15:03:47 -05:00
evazion bd47641601 twitter: don't fail when api key isn't configured. 2018-09-16 15:03:47 -05:00
evazion 325120ee51 twitter: fix parsing of the artist name from the url.
Fixes URLs like https://twitter.com/intent/user?user_id=123 being
incorrectly normalized to http://twitter.com/intent/ in artist entries.

Also fixes the artist name to be taken from the url when it can't be
obtained from the api (when the tweet is deleted).
2018-09-16 15:03:23 -05:00
evazion c9b3c8d217 artist urls: add tests for controller + artist url search. 2018-09-15 19:58:55 -05:00
evazion 761f2649af artists: add more invalid name tests (#3901). 2018-09-14 13:32:45 -05:00
Albert Yi f487b2a2c6 Merge pull request #3889 from evazion/fix-replace-artist-finder
Cleanup artist finder
2018-09-12 11:44:42 -07:00
evazion fbd5f6b7f2 pixiv: fix preview_urls for ugoiras (#3891). 2018-09-12 00:43:10 -05:00
evazion 37fc215d75 pixiv: fix preview_urls to use correct url (#3891). 2018-09-11 23:55:46 -05:00
evazion 583f8457f0 artists: clean up artist finding logic.
Rename Artist#find_all_by_url to url_matches and drop previous
url_matches method, along with find_artists and search_for_profile.

Previously find_artists tried to lookup the url, referer url, and profile
url in turn until an artist match was found. This was wasteful, because
the source strategy already knows which url to lookup (usually the profile
url). If that url doesn't find a match, then the artist doesn't exist.
2018-09-11 20:14:46 -05:00
Albert Yi a5df178bcc Merge pull request #3886 from r888888888/source-api-caching
cache api clients
2018-09-11 17:34:25 -07:00
Albert Yi 4972c998f8 rely on preview urls if available for gallery 2018-09-11 15:06:12 -07:00
Albert Yi 266c7c0d5b cache api clients 2018-09-11 14:19:17 -07:00
Albert Yi f16c3a3f40 fix nijie specs 2018-09-11 13:27:00 -07:00
evazion 9a980367f6 twitter: normalize artist commentaries to nfkc (#3719)
Fixes hashtags not being interpreted when the author uses a fullwidth
number sign (#, U+FF03).

ref: https://github.com/r888888888/danbooru/issues/3719#issuecomment-419535610
2018-09-10 21:45:50 -05:00
evazion c9300cc54e sta.sh: add tests + docs. 2018-09-10 19:26:53 -05:00
evazion 0d7b7e6e85 twitter tests: fix dead tweet. 2018-09-09 14:10:43 -05:00
evazion b924c2bb9c nijie: fix artist url normalization. 2018-09-09 13:17:52 -05:00
evazion a67edb8783 deviantart: fix artist finder for artist names with underscores.
Fix the artist finder for urls like this:

  https://orig00.deviantart.net/4274/f/2010/230/8/a/pkmn_king_and_queen_by_mikoto_chan.jpg

that don't contain a deviantart id but do contain the artist name.
2018-09-07 12:23:48 -05:00
evazion 610391205f deviantart: fix artist finder for profile urls missing the 'www'.
Fix the artist finder to work when the profile url in the artist entry
is missing the 'www'. Example:

  https://deviantart.com/noizave
  https://www.deviantart.com/noizave
2018-09-07 11:36:48 -05:00
evazion aee1906761 Fix #3738: Artist URL search should be case-insensitive for domains. 2018-09-05 19:14:24 -05:00
Albert Yi 69b7c9efd0 Merge pull request #3838 from evazion/fix-3837
Fix #3837: Remove expired bans.
2018-09-05 12:03:46 -07:00
Albert Yi a41d006380 fix pool ordering js (#3869), fix forum topic tests 2018-09-05 12:01:59 -07:00
Albert Yi 314341773c fix tests 2018-09-04 18:23:49 -07:00
Albert Yi a5943de418 Merge pull request #3868 from evazion/fix-3864
Fix #3864: DeviantArt fetch source data failure
2018-09-04 13:42:01 -07:00
Albert Yi 8ec96f42f7 fix specs 2018-09-04 13:38:09 -07:00