Commit Graph

377 Commits

Author SHA1 Message Date
evazion
f38c38f26e search: split tag_match into user_tag_match / system_tag_match.
When doing a tag search, we have to be careful about which user we're
running the search as because the results depend on the current user.
Specifically, things like private favorites, private favorite groups,
post votes, saved searches, and flagger names depend on the user's
permissions, and whether non-safe or deleted posts are filtered out
depend on whether the user has safe mode on or the hide deleted posts
setting enabled.

* Refactor internal searches to explicitly state whether they're
  running as the system user (DanbooruBot) or as the current user.
* Explicitly pass in the current user to PostQueryBuilder instead of
  implicitly relying on the CurrentUser global.
* Get rid of CurrentUser.admin_mode? (used to ignore the hide deleted
  post setting) and CurrentUser.without_safe_mode (used to ignore safe
  mode).
* Change the /counts/posts.json endpoint to ignore safe mode and the
  hide deleted posts settings when counting posts.
* Fix searches not correctly overriding the hide deleted posts setting
  when multiple status: metatags were used (e.g. `status:banned status:active`)
* Fix fast_count not respecting the hide deleted posts setting when the
  status:banned metatag was used.
2020-05-07 03:29:44 -05:00
nonamethanks
804c87a0ea Artstation: artists can have underscores in their name 2020-04-17 01:10:23 +02:00
lllusion3469
65fb0cf510 tumblr: pick biggest image based on resolution
photo[:alt_sizes] may contain a bigger image than
photo[:original_size]

fixes #4324
2020-04-08 16:42:17 +02:00
BrokenEagle
a45ae09d72 Account for additional Twitter video image links 2020-03-29 19:27:05 +00:00
BrokenEagle
9e16c01285 Allow video thumbnails as direct Twitter images 2020-03-20 20:59:37 +00:00
evazion
ddffffb413 artists: factor out artist finder to separate module. 2020-03-06 23:23:38 -06:00
evazion
49a3538933 pixiv: add support for techorus urls. 2020-03-04 00:00:39 -06:00
evazion
266e4054b0 Fix #4293: ArtStation: use 4k images.
Also fixes #4290 (Image replacements: undefined method hostname for nil:NilClass)
2020-03-03 23:01:29 -06:00
evazion
1244e02fe2 pixiv: handle new https://i-f.pximg.net urls. 2020-02-18 19:22:57 -06:00
BrokenEagle
0569e8346c Fix profile url for normalization when Pawoo errors 2020-01-29 22:27:10 +00:00
evazion
60bf21ff80 twitter: fix preview_urls when source url is a direct image.
Fix preview_urls returning an empty array when the source url is a
direct image from Twitter.

Also return preview_urls in /source.json.
2020-01-21 16:34:03 -06:00
evazion
e42881fbbf Fix #4262: Exception when using Twitter video image links. 2020-01-15 15:20:33 -06:00
evazion
aff3d3b18f Fix various rubocop issues. 2020-01-11 19:01:40 -06:00
BrokenEagle
114637f594 Use more efficient use of regexes for Twitter hashtag normalization 2020-01-10 15:30:54 +00:00
BrokenEagle
2eef73e8fe Add additional Twitter hashtag
- It was mentioned on issue #4047 but was forgotten to be included
2020-01-10 14:52:49 +00:00
evazion
faeec18efc twitter: add hashtag normalization test.
Add test for #4243. Also fix warning from bootsnap:

    iseq.rb:18: warning: nested repeat operator '+' and '?' was replaced with '*' in regular expression: /(?<!\A)生誕祭(?:\d+)?\z/
2020-01-05 17:38:30 -06:00
BrokenEagle
67860a845c Add normalization for Twitter hashtags 2020-01-05 23:01:50 +00:00
evazion
309821bf73 rubocop: fix various style issues. 2019-12-22 21:23:37 -06:00
evazion
84ba1d417f Fix #4220: Uploading from Tumblr is broken. 2019-12-15 19:04:52 -06:00
evazion
41378bc8e3 sources: replace HttpartyCache with Danbooru::Http. 2019-12-15 17:06:58 -06:00
evazion
da84e3a2f2 twitter: replace twitter gem with our own API client.
The twitter gem had several problems:

* It's been unmaintained for over a year.
* It pulled in a lot of dependencies, many of which were outdated. In
  particular, it locked the `http` gem to version 3.3, preventing us
  from upgrading to 4.2.
* It raised exceptions on normal error conditions, like for deleted
  tweets or suspended users, which we really don't want.
* We had to wrap it to provide caching.

Changes:

* Fixes #4226 (Exception when creating new artists entries for suspended
  Twitter accounts)
* Drop support for scraping images from summary cards. Summary cards
  are the previews you get when you link to a website in a tweet. These
  preview images aren't always the best image.
2019-12-13 17:27:03 -06:00
evazion
1c3b96dc4e deviantart: fix non-downloadable posts (#4219).
Adapted from e51e1aa7dd.
2019-12-06 22:54:08 -06:00
evazion
a15bbe4264 moebooru: fix preview_urls to fallback to image_urls.
Fix the moebooru strategy to fallback to returning the image url if we
can't find the preview url. Fixes iqdb lookups failing in some cases
because the strategy didn't return a valid url for preview_url.
2019-10-26 13:51:58 -05:00
evazion
8209a75e95 nicoseiga: remove referer spoofing.
NicoSeiga doesn't appear to have any hotlink protection, so we don't
need to spoof the referer.
2019-10-07 13:15:48 -05:00
evazion
03d9b3feca pixiv: support new https://www.pixiv.net/artworks/:id urls. 2019-09-24 03:33:21 -05:00
evazion
1b426fb23f Fix #4150: Nijie strategy fails for mp4 files. 2019-09-03 22:33:09 -05:00
evazion
64eb6dbb2a pixiv: possible fix for #4152. 2019-09-02 13:13:58 -05:00
evazion
e781c6b608 pixiv: temp disable source strategy (#4152).
Disable use of the Pixiv API until we get it working again.
2019-09-02 11:13:00 -05:00
evazion
eba6440b8b Fix #4144: Deviantart Eclipse update broke strategy. 2019-08-28 23:40:29 -05:00
evazion
7f482dc35b deviantart: normalize wixmp.com sources to page urls.
Normalize sources like this:

   https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/f/ab917938-d8c1-4b58-933d-0d38c390461f/ddcyi98-cbd39da2-f528-4b26-aadb-a16fe91442b2.jpg/v1/fill/w_1280,h_1760,q_100,strp/tifa_by_chubymi_ddcyi98-fullview.jpg?token=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJzdWIiOiJ1cm46YXBwOjdlMGQxODg5ODIyNjQzNzNhNWYwZDQxNWVhMGQyNmUwIiwiaXNzIjoidXJuOmFwcDo3ZTBkMTg4OTgyMjY0MzczYTVmMGQ0MTVlYTBkMjZlMCIsIm9iaiI6W1t7ImhlaWdodCI6Ijw9MTc2MCIsInBhdGgiOiJcL2ZcL2FiOTE3OTM4LWQ4YzEtNGI1OC05MzNkLTBkMzhjMzkwNDYxZlwvZGRjeWk5OC1jYmQzOWRhMi1mNTI4LTRiMjYtYWFkYi1hMTZmZTkxNDQyYjIuanBnIiwid2lkdGgiOiI8PTEyODAifV1dLCJhdWQiOlsidXJuOnNlcnZpY2U6aW1hZ2Uub3BlcmF0aW9ucyJdfQ.Uochlma4QJmLwL2ZGMmvTr4HMva4m4bCeF3vnyPSw4I

to this:

   https://www.deviantart.com/chubymi/art/Tifa-807825644

on the sidebar of the posts show page.
2019-08-06 12:04:05 -05:00
evazion
a8896b664d twitter: fix batch bookmarklet selecting wrong image.
Fix regression in 7e465aeda.

https://danbooru.donmai.us/forum_topics/9127?page=276#forum_post_158779
2019-08-06 10:42:45 -05:00
evazion
7e465aedae Fix #4110: New Twitter image urls are broken in bookmarklet. 2019-08-04 20:23:10 -05:00
evazion
9ecf36585c nijie: update for new image urls.
Nijie moved from this:

    https://pic03.nijie.info/nijie_picture/236014_20170620101426_0.png (page: https://www.nijie.info/view.php?id=218856)

to this:

    https://pic.nijie.net/03/nijie_picture/236014_20170620101426_0.png (page: https://www.nijie.info/view.php?id=218856)
2019-08-04 17:49:54 -05:00
Albert Yi
8a16e089d2 don't try to parse deviant art urls that lack a deviation id in the url 2019-04-16 14:00:38 -07:00
Albert Yi
6f4ca75bb1 try to handle deivantart images hosted on images-wixmp better 2019-04-16 13:55:26 -07:00
Albert Yi
eb8ce2e955 Revert "eliminate old algorithm for deviant art sources, fix canonical_url on deviantart"
This reverts commit 58ca52b43e.
2019-04-16 13:22:09 -07:00
Albert Yi
58ca52b43e eliminate old algorithm for deviant art sources, fix canonical_url on deviantart 2019-04-12 16:35:41 -07:00
Albert Yi
d8d4a5ae6f refactor nico seiga manga support 2019-02-25 15:53:07 -08:00
Albert Yi
90ce42a537 add support for nico seiga manga (fixes #4060) 2019-02-25 14:44:45 -08:00
evazion
13dff046f7 pixiv: fix illust id parsing (fixup 8cadef2dd) 2019-01-13 15:02:51 -06:00
evazion
8cadef2dd7 pixiv: fix illust id parsing (fix #4043).
* Tighten up illust id parsing to avoid misparsing ids from
  non-illust urls (sketch urls and novel urls).

* Move id parsing tests from post_test.rb to sources/pixiv_test.rb.

* Drop support for touch.pixiv.net urls. These urls are no longer used
  by Pixiv and aren't present as the source of any posts on Danbooru.
2019-01-13 14:28:51 -06:00
evazion
04d5b16da7 pixiv: fix failure to upload bad pixiv id images (fix #4031)
Bug: Uploading bad pixiv id images failed because the pixiv strategy
raised a BadIDError exception when the upload service checked for the
ugoira frame data.
2019-01-03 18:01:20 -06:00
evazion
b12e8b5058 Fix #4030: Artstation artist_name broken for direct image urls. 2018-12-31 11:45:50 -06:00
evazion
0f513d1a1b twitter: include intent url in new artist entries (#4028). 2018-12-27 15:03:11 -06:00
evazion
2129e60b2b pixiv: include stacc url in new artist entries (#4028). 2018-12-27 15:03:11 -06:00
evazion
1f73e60514 sources: add methods for customizing new artist entries.
* Rename `unique_id` to `tag_name`.

* Add `other_names` and `profile_urls` methods that sources can override
  to provide extra names or urls when creating new artist entries.
2018-12-27 15:03:11 -06:00
evazion
2170961f47 artists: improve prefilling of new artist form (#4028)
* When creating an artist by clicking the '?' next to the artist tag in
  the tag list, prefill the new artist form by finding the artist's last
  upload and fetching its source data.

  Previously we filled the urls with the source of the artist's last
  upload, which was wrong because it was usually a direct image URL (#3078).

* Fix the other names field not escaping spaces within names to underscores.

* Fix the other names field being potentially prefilled with duplicate names.
2018-12-27 15:03:11 -06:00
Albert Yi
ca7c56f95f fix: url normalization for non-project artist subdomains (maybe fixes #4021) 2018-12-19 15:19:00 -08:00
evazion
c700ea4b5f Fix #4016: Translated tags failing to find some tags.
* Normalize spaces to underscores when saving other names. Preserve case
  since case can be significant.

* Fix WikiPage#other_names_include to search case-insensitively (note:
  this prevents using the index).

* Fix sources to return the raw tags in `#tags` and the normalized tags
  in `#normalized_tags`. The normalized tags are the tags that will be
  matched against other names.
2018-12-16 11:37:57 -06:00
evazion
7bbe4e8ac1 moebooru: don't exclude artists from translated tags (#3969). 2018-12-05 11:53:26 -06:00