jaculabilis/danbooru
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
7,489 Commits 1 Branch 0 Tags
39f9e01b131caca20ff8b2fe532007428c154036
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
evazion 39f9e01b13 nicoseiga: fix canonical_url to use the image url. 2018-09-22 11:07:18 -05:00
evazion 09a8198979 /artists: add wildcard, regex search to url field (#3900) 
Allow searching the URL field by regex or by wildcard.

If the query looks like `/twitter/` do a regex search, otherwise if it
looks like `http://www.twitter.com/*` do a wildcard search, otherwise if
it looks like an url do an artist finder search, lastly if it looks like
`twitter` do a `*twitter*` search.
 2018-09-21 21:19:01 -05:00
evazion a4608daf38 /artists: add more search options for other names, group name. 
Add these search params:

* /artists?search[<field>]=
* /artists?search[<field>_eq]=
* /artists?search[<field>_not_eq]=
* /artists?search[<field>_like]=
* /artists?search[<field>_not_like]=
* /artists?search[<field>_ilike]=
* /artists?search[<field>_not_ilike]=
* /artists?search[<field>_regex]=
* /artists?search[<field>_not_regex]=

where `<field>` can be `name`, `group_name`, or `other_names`.

Remove these search params:

* /artists?search[name_matches]=
* /artists?search[other_names_match]=
* /artists?search[group_name_matches]=

`/artists?search[<field>_like]=` effectively does the same thing that
these searches did.
 2018-09-21 20:55:14 -05:00
evazion f917b83d6f /artists: drop deprecated search syntax, add regex search for names. 
Drop support for the following pseudo-metatags in the Name field in the
artists search form:

* name:<name>
* other:<other name>
* group:<group name>
* status:banned
* status:active
* http://www.example.com

Instead, make the Name field do a wildcard search against the artist
name, group name, or other names. If the query looks like `/regex/`,
then do a regex search against any of these names.

/artists?search[name] now does a literal exact match and
/artists?search{any_name_matches] does the above wildcard/regex search.
 2018-09-21 20:51:53 -05:00
evazion 237ab9b782 dmail filters: fix filters being case sensitive. 2018-09-20 19:51:59 -05:00
evazion 03abbd0683 Fix #2894: Use [[:space:]] instead of \s in regexes. 2018-09-20 19:24:38 -05:00
evazion 29cdaddd86 PostSetPresenters::Post#related_posts: clean up metatag parsing (#2894). 
* Fix `#related_tags` to use `Tag.has_metatag?`.
* Fix Tag::SUBQUERY_METATAGS and Tag::METATAGS to be arrays instead of regexes.
 2018-09-20 19:23:47 -05:00
evazion 6fe883c316 posts index: clean up limit:<n> / order:random parsing (#2894) 
* Move the limit:<n> / order:random metatag parsing from the controller
  to the post set.

* Introduce `Tag.has_metatag?` and use it to parse these metatags
  instead of using a regex (#2894).
 2018-09-20 19:23:47 -05:00
evazion 235271706b Fix #3913: Remove legacy /post/create.xml endpoint. 2018-09-20 12:25:16 -05:00
evazion 958a9f505b moebooru: rewrite sample urls + support bookmarklet on html page. 
* Fixes #2942: Add Moebooru Rewrite for Sample Images.
* Addresses #3911: Improve Moebooru support.
 2018-09-19 23:32:21 -05:00
evazion f4e08ef30d Downloads::File: fix SSRF inside is_cloudflare? (#2498). 
Fixes the banned IP check not being applied when sending the HEAD
request for is_cloudflare?.

Also fixes the `#size` method not using the uncached url (which meant
the bookmarklet could report the wrong filesize on artstation uploads).
 2018-09-19 20:11:53 -05:00
evazion 2f17082e73 Downloads::File: fix SSRF when following redirects (#2498). 
Fixes the banned IP check not being applied when following redirects:

  http://danbooru.donmai.us/uploads/new?url=http://httpbin.org/redirect-to%3Furl=http://127.0.0.1/test.jpg
 2018-09-18 16:00:03 -05:00
evazion 99221e4028 Downloads::File: fix SSRF attack when fetching remote size (#2498). 
Fixes the banned IP check not being applied when fetching the remote
file size. This allowed one to trick Danbooru into sending HEAD requests
to private IPs:

  http://danbooru.donmai.us/uploads/new?url=http://127.0.0.1/test.jpg
 2018-09-18 12:16:27 -05:00
evazion 488b1fd994 Downloads::File: memoize strategy. 
Also remove unused options param from constructor
 2018-09-18 10:22:15 -05:00
evazion 9cdfbba6c2 Fix #3910: Corrupted images during upload. 
Use a fresh tempfile for each download attempt instead of reusing the same
file (and having to rewind/truncate it after each failed attempt).
 2018-09-18 10:01:44 -05:00
evazion d3c135ec72 Downloads::File#http_get_streaming: clean up retry logic. 
Replace handrolled retry logic with retriable gem (already pulled in by another gem).
 2018-09-18 09:44:15 -05:00
evazion 96e89cecfb tests: move twitter canonical url test. 2018-09-17 23:27:53 -05:00
evazion 5f328c842a /artist_urls: fix timestamp formatting. 2018-09-17 21:01:42 -05:00
evazion 823f78af49 autocomplete.js: add data attrs on autocomplete results (#3902). 2018-09-17 18:18:33 -05:00
evazion 2d5f6b8a35 Fix #3902: Add source of tag autocomplete to the results. 2018-09-17 18:07:48 -05:00
evazion 4a99cb098f moebooru: use the image url as the canonical url. 2018-09-16 21:00:11 -05:00
evazion d9ce953752 Fix #3906: Moebooru strategy raises NotImplementedError. 2018-09-16 21:00:11 -05:00
evazion cae78fa8ee moebooru: move tests from unit/downloads to unit/sources. 2018-09-16 21:00:11 -05:00
evazion f135a7c064 twitter: normalize canonical urls. 
Normalize http://mobile.twitter.com to http://twitter.com in canonical urls.
 2018-09-16 15:03:47 -05:00
evazion bd47641601 twitter: don't fail when api key isn't configured. 2018-09-16 15:03:47 -05:00
evazion 325120ee51 twitter: fix parsing of the artist name from the url. 
Fixes URLs like https://twitter.com/intent/user?user_id=123 being
incorrectly normalized to http://twitter.com/intent/ in artist entries.

Also fixes the artist name to be taken from the url when it can't be
obtained from the api (when the tweet is deleted).
 2018-09-16 15:03:23 -05:00
evazion 4a84ec9e26 artist urls: add trigram indexes on urls. 2018-09-15 19:58:55 -05:00
evazion c9b3c8d217 artist urls: add tests for controller + artist url search. 2018-09-15 19:58:55 -05:00
evazion 3afc0b3a78 artist urls: add more url search params for /artist_urls. 
Adds these search params:

* /artist_urls?search[url]=...
* /artist_urls?search[url_eq]=...
* /artist_urls?search[url_not_eq]=...
* /artist_urls?search[url_like]=...
* /artist_urls?search[url_ilike]=...
* /artist_urls?search[url_not_like]=...
* /artist_urls?search[url_not_ilike]=...
* /artist_urls?search[url_regex]=...
* /artist_urls?search[url_not_regex]=...

and likewise for normalized_url.
 2018-09-15 19:58:54 -05:00
evazion c06af060f9 artist urls: add artist, url_matches search params to /artist_urls. 2018-09-15 19:58:31 -05:00
evazion 1fce794b99 artist urls: add /artist_urls index page. 2018-09-15 19:58:05 -05:00
evazion 761f2649af artists: add more invalid name tests (#3901). 2018-09-14 13:32:45 -05:00
evazion bbb233b93f css: remove unused or redundant rules. 2018-09-14 13:31:59 -05:00
evazion 0aecc9d34c css: fix 020_base.scss getting imported twice. 
This import caused all in the rules in 020_base.scss to be included
twice in the final stylesheet.
 2018-09-14 10:31:30 -05:00
evazion a5ef97a608 css: remove obsolete/unused mixins. 2018-09-14 00:34:34 -05:00
evazion 42f97bf3e5 Fix external link icons (#3890). 2018-09-14 00:21:08 -05:00
evazion 26674ec88a font awesome: switch from js+svg to css+webfont version (fix #3890). 2018-09-14 00:18:30 -05:00
evazion de78c5fd68 /recommended_posts: enable blacklists. 2018-09-13 23:06:24 -05:00
evazion 785fcda79f /recommended_posts: fix inconsistency in controller/action ids. 2018-09-13 23:06:10 -05:00
evazion 091c4c97a8 Fix #3895: 'Related tags' not appearing automatically on uploads page. 2018-09-12 19:26:59 -05:00
evazion fbd5f6b7f2 pixiv: fix preview_urls for ugoiras (#3891). 2018-09-12 00:43:10 -05:00
evazion 0df5937d33 /uploads/batch: use preview_url for non-proxied images (#3891). 2018-09-12 00:08:28 -05:00
evazion 37fc215d75 pixiv: fix preview_urls to use correct url (#3891). 2018-09-11 23:55:46 -05:00
evazion d5c7de84a3 font awesome: disable data-search-pseudo-elements (#3890). 2018-09-11 21:20:24 -05:00
evazion 583f8457f0 artists: clean up artist finding logic. 
Rename Artist#find_all_by_url to url_matches and drop previous
url_matches method, along with find_artists and search_for_profile.

Previously find_artists tried to lookup the url, referer url, and profile
url in turn until an artist match was found. This was wasteful, because
the source strategy already knows which url to lookup (usually the profile
url). If that url doesn't find a match, then the artist doesn't exist.
 2018-09-11 20:14:46 -05:00
evazion 10ca4dd3ad artists: replace artist finder with fetch source data. 
* On posts, automatically trigger "Fetch source data" when clicking the
  Edit tab, instead of triggering the artist finder button. This way we
  find both the artist and the translated tags in one ajax call.

* Remove the "Artist" finder button next to the source field. This isn't
  necessary given that "Fetch source data" finds the artist itself.

* Remove the /artists/finder.json API endpoint. This is no longer used
  after removing the "Artist" finder button.
 2018-09-11 20:09:25 -05:00
evazion 8915437b4a Fix #3439: Identify external links in DText. 2018-09-11 15:10:59 -05:00
evazion 9a980367f6 twitter: normalize artist commentaries to nfkc (#3719) 
Fixes hashtags not being interpreted when the author uses a fullwidth
number sign (＃, U+FF03).

ref: https://github.com/r888888888/danbooru/issues/3719#issuecomment-419535610
 2018-09-10 21:45:50 -05:00
evazion a0ebd90409 Post#normalized_source: add hitomi.la url normalization. 2018-09-10 20:22:10 -05:00
evazion bfed323988 deviantart: fix page_url for when api data is unavailable. 
The api data is unavailable when we can't scrape the uuid, either
because the work is deleted or because the work is actually a sta.sh
upload and we weren't given the sta.sh page in the referer url.
 2018-09-10 19:26:53 -05:00
evazion c9300cc54e sta.sh: add tests + docs. 2018-09-10 19:26:53 -05:00
evazionandlllusion3469 7d5d098636 Fix #3877: Add sta.sh strategy. 
Co-authored-by: lllusion3469 <31420484+lllusion3469@users.noreply.github.com>
 2018-09-10 19:26:47 -05:00
evazion 10358d4211 Fix #3881: It has been reviewed by X moderators.“ message. 2018-09-10 17:39:49 -05:00
evazion de10ea66a1 <role>_only: fix role checking logic to check ip bans last. 
Make <role>_only methods check the role first and ip bans last. This
avoids hitting the database for anonymous users, since they'll always
fail the is_<role>? check before the ip check.
 2018-09-09 21:53:24 -05:00
evazion e546e52bd7 Replace AnonymousUser with User.new. 
* Replace AnonymousUser null object with a readonly, unpersisted User object.

* Default always_resize_images to true (previously it was true for
  anonymous users, but false for new members).

* Default comment_threshold to -1 for anonymous users (previously it was
  0 for anonymous but -1 for new members).
 2018-09-09 21:53:08 -05:00
evazion cb2d85d925 twitter: fix profile_url for twitter.com/i/web/status/:id urls. 
Fix profile_url returning nil for https://twitter.com/i/web/status/943446161586733056.
 2018-09-09 19:48:34 -05:00
evazion 0d7b7e6e85 twitter tests: fix dead tweet. 2018-09-09 14:10:43 -05:00
evazion b924c2bb9c nijie: fix artist url normalization. 2018-09-09 13:17:52 -05:00
evazion 056fe365f2 Render "Fetch source data" box html server-side. 
Instead of calling /sources.json and rendering the "Fetch source data"
box client-side in Javascript, call /sources.js so we can render the
html server-side.
 2018-09-08 15:42:16 -05:00
evazion 99a5e885e0 artist_url.rb: remove legacy artist url normalization code. 
* `legacy_normalize` came from c6012535, which is no longer a problem.

* `normalize_for_search` is only used for "[mass edit]" links
in artist entries. These links are a shortcut for performing a
`-artist_name source:<artist_url> -> artist_name` mass edit to tag
untagged artists, but this won't work for most sites these days.
 2018-09-07 12:55:51 -05:00
evazion 8e7dd9e97f artist_url.rb: remove unnecessary deviantart profile url normalization. 
This is now handled in the source strategy.
 2018-09-07 12:26:50 -05:00
evazion a67edb8783 deviantart: fix artist finder for artist names with underscores. 
Fix the artist finder for urls like this:

  https://orig00.deviantart.net/4274/f/2010/230/8/a/pkmn_king_and_queen_by_mikoto_chan.jpg

that don't contain a deviantart id but do contain the artist name.
 2018-09-07 12:23:48 -05:00
evazion 610391205f deviantart: fix artist finder for profile urls missing the 'www'. 
Fix the artist finder to work when the profile url in the artist entry
is missing the 'www'. Example:

  https://deviantart.com/noizave
  https://www.deviantart.com/noizave
 2018-09-07 11:36:48 -05:00
evazion 22c27f8910 post previews: fix css for captioned post previews. 
Captioned post previews (previews with the pool name, similarity, or
size beneath) need to have `height: auto` set, otherwise they'll default
to `height: 154px` (just enough for the image) and the caption won't be visible.
 2018-09-06 20:44:47 -05:00
evazion 950fcdb7b2 uploads: add new source:<url> dupe check (fix #3873) 
* On the /uploads/new page, instead of just showing a "This post has
probably already been uploaded" message, show the actual thumbnails of
posts having the same source as what the user is trying to upload.

* Move the iqdb results section up top, beside the related posts section.
 2018-09-06 20:43:20 -05:00
evazion 07c0695ac1 uploads: remove "this post was probably already uploaded" dupe check. 
Also makes it so that using the bookmarklet always triggers async upload
preprocessing. Before it was only triggered when the source passed a
dupe check, but that check was inaccurate (#3873).
 2018-09-06 20:43:04 -05:00
evazion 06cf32160b user.rb: memoize upload limit methods. 
These methods get called multiple times when rendering UserPresenter#upload_limit
on the upload page. This caused repeated COUNT(*) queries.
 2018-09-06 20:31:51 -05:00
evazion 7ecf6f1ba2 Fix link colors inside dialogs, autocomplete. 
ref: f9f0da054, https://github.com/r888888888/danbooru/issues/3791#issuecomment-410378703.
 2018-09-05 23:48:51 -05:00
evazion aee1906761 Fix #3738: Artist URL search should be case-insensitive for domains. 2018-09-05 19:14:24 -05:00
evazion c2cf6a5441 Fix #3871: Autocomplete can return duplicate tags. 2018-09-05 18:56:30 -05:00
evazion 6c457c86ac nav menu: fix 'Forum' link not getting '.current' class. 
Fixes the '.current' class not being set on the 'Forum' link when
viewing the /forum_topics page. As a result, the link wasn't bolded and
didn't have the background set.
 2018-09-04 18:15:21 -05:00
evazion 5c457fbe51 pixiv: remove obsolete edgesuite.net rewrite rule. 
This CDN hasn't been seen for several years.

ref: https://danbooru.donmai.us/forum_topics/10766
 2018-09-04 18:15:21 -05:00
evazion 4bbe09762d pixiv: remove dead methods (#is_manga?, #page_count, #page). 2018-09-04 18:15:21 -05:00
evazion 3bed5f3cf9 Fix #3867: Add namespaces to all Javascript events. 2018-09-04 12:26:18 -05:00
evazion c03af7f563 posts.js: remove dead search dropdown code. 2018-09-04 12:26:18 -05:00
evazion 1bfe763182 related tags: remove dead #artist-tags-container code. 
Dead since 78900c4d3.
 2018-09-03 19:13:09 -05:00
evazion e37844303d deviantart: take artist name from url when unavailable from API. 
In some cases we can't get the artist name from the API, either because
we can't do the API call because the url doesn't contain a deviation id,
or because the work is deleted:

* http://fc08.deviantart.net/files/f/2007/120/c/9/cool_like_me_by_47ness.jpg (work: http://fav.me/dwcohb)
* https://pre00.deviantart.net/423b/th/pre/i/2017/281/e/0/mindflayer_girl01_by_nickbeja-dbpxdt8.png (work: http://fav.me/dbpxd58)

Switch to taking the artist name from the url (when present) to deal
with these cases. Fixes the artist finder and the artist url normalizer
to work in this situation.
 2018-09-03 18:27:01 -05:00
evazion 8f87fb90d9 deviantart: handle urls without deviation ids (fix #3864) 
Some older URL formats don't contain the deviation id:

* http://fc08.deviantart.net/files/f/2007/120/c/9/Cool_Like_Me_by_47ness.jpg
* http://pre06.deviantart.net/8497/th/pre/f/2009/173/c/c/cc9686111dcffffffb5fcfaf0cf069fb.jpg

In these cases we can't make the API call. Fix failures due to not being
able to do API calls in this situation.

Also fix canonical_url to use the image_url when it contains the
deviation id, or the page_url when it doesn't.

Finally, fix page_url to use the url from the API instead of the raw url
given by the user, so that it's in a consistent form for canonical_url.
 2018-09-03 18:26:45 -05:00
evazion 316acead16 deviantart: fix error when uploading image belonging to deleted work. 2018-09-02 23:09:40 -05:00
evazion 2d1b1311d6 deviantart: fix sample urls not being rewritten to full size urls. 2018-09-02 23:09:29 -05:00
evazion 807c3dd5f4 deviantart: remove obsolete image sample rewrite rules. 
Remove rewrite rules for these types of sample urls:

* http://th00.deviantart.net/fs71/PRE/f/2014/065/3/b/goruto_by_xyelkiltrox-d797tit.png
* http://th04.deviantart.net/fs70/300W/f/2009/364/4/d/Alphes_Mimic___Rika_by_Juriesute.png

These URLs aren't served to users any more, and just stripping out "PRE"
or "200H" isn't sufficient to get the full size image. In general, an
api call is required to find the full size image url.
 2018-09-02 14:49:58 -05:00
evazion b9ed676bfb deviantart: handle origin-orig.deviantart.net urls. 2018-09-02 13:57:15 -05:00
evazion 6fc2d17886 Post#normalized_source: normalize da urls to new format (#3771) 2018-09-02 11:19:15 -05:00
evazion d693f01dde Fix #3859: Related tag and find artist don't run when fetch data fails. 
Fixes an exception in the artist finder caused by searching for a nil profile_url.
 2018-09-01 11:48:42 -05:00
evazion d6235d6f9e nijie: add canonical url tests. 2018-08-31 23:23:15 -05:00
evazion bc7ead5396 Fix #3857: Add searching mod actions by message. 
Also allow searching user feedbacks and bulk update requests.
 2018-08-31 19:59:51 -05:00
evazion 0eff095a3e Refactor searching text attributes. 
* Allow using ApplicationRecord#attribute_matches to search text attributes,
and standardize models on using this instead of duplicating code.

* Remove restrictions that limited wildcard searches to Builders only in various places.
 2018-08-31 19:50:46 -05:00
evazion 736c22c3ce pixiv: fix fetch source data failing on fanbox images. 
ref: https://danbooru.donmai.us/forum_posts/150000
 2018-08-31 00:21:03 -05:00
evazion c689a161f6 pixiv: fix failure when normalizing pixiv stacc artist urls (#3856). 2018-08-30 19:24:44 -05:00
evazion a68d12bd45 autocomplete: fix case sensitivity (#3854). 
ref: https://github.com/r888888888/danbooru/issues/3854#issuecomment-417366253:
 2018-08-30 17:38:29 -05:00
evazion 6c94047556 Sources::Strategies::Twitter#profile_url: fix case when url is a profile url. 2018-08-29 19:29:16 -05:00
evazion a1044dbc19 twitter: fix handling of direct image urls without a referer url. 2018-08-29 17:14:57 -05:00
evazion cd11ae6008 upload page: fix fetch source data when not using bookmarklet. 
* #upload_referer_url no longer exists, just using #ref works.
* #upload_source defaults to params[:url], so using it instead of #url still works.

ref: https://github.com/r888888888/danbooru/issues/3850#issuecomment-417116385
 2018-08-29 16:54:03 -05:00
evazion bf19ea3bd1 twitter: fix typo in ASSET regex (#3850). 2018-08-29 15:09:36 -05:00
evazion a8ae4d3f36 Fix #3849: Forum Button in the topbar has broken css 2018-08-29 14:29:15 -05:00
evazion 4f5cb0ba70 Post#has_artist_tag: fix "artist tag is required" triggering incorrectly. 
Fixes the "Artist tag is required" upload warning to only trigger when
the source is from a first-party site.
 2018-08-28 18:14:21 -05:00
evazion 604bfb0923 Fix #3846: Subscribed forum posts link to invalid URL in email updates. 
Test: http://localhost/rails/mailers/user_mailer/forum_notice
 2018-08-27 22:04:37 -05:00
evazion 47e26419d3 nav menu: add html ids to secondary nav menu links (#3844). 2018-08-27 20:57:59 -05:00
evazion 4e1d76da81 html: add controller & action attributes to <body> tag (#3844). 2018-08-26 16:02:06 -05:00
evazion b98bc26397 nav menu: add html ids to main menu nav links (#3844). 2018-08-26 13:12:41 -05:00