Commit Graph

64 Commits

Author SHA1 Message Date
evazion
07e23204b6 rubocop: fix various Rubocop warnings. 2021-06-17 04:17:53 -05:00
nonamethanks
e75489aa94 Twitter: update common hashtag regexes 2021-03-02 17:05:47 +01:00
BrokenEagle
c90ef9f1b0 Add the Twitter name to the list of other names for new artists
The artist name is supposed to be the display name according to the
base file, however the artist name was treated like the tag name
instead. This commit renames all instances of "artist_name" to
"tag_name" and then adds an "artist_name" function that uses the
Twitter display name if available.
2021-01-19 00:46:21 +00:00
nonamethanks
79a59e52ec Twitter: don't get api without a status id 2020-06-20 15:08:49 +02:00
evazion
1aa0f65187 sources: fix rubocop warnings. 2020-06-16 00:10:37 -05:00
nonamethanks
307df3b3e4 Refactor source normalization
* Move the source normalization logic out of the post model
  and into individual sources' strategies.
* Rewrite normalization tests to be handled into each source's test,
  and expand them significantly. Previously we were only testing
  a very small subset of domains and variants.
* Fix up normalization for several sites.
* Normalize fav.me urls into normal deviantart urls.
2020-05-21 22:46:51 +02:00
BrokenEagle
a45ae09d72 Account for additional Twitter video image links 2020-03-29 19:27:05 +00:00
BrokenEagle
9e16c01285 Allow video thumbnails as direct Twitter images 2020-03-20 20:59:37 +00:00
evazion
60bf21ff80 twitter: fix preview_urls when source url is a direct image.
Fix preview_urls returning an empty array when the source url is a
direct image from Twitter.

Also return preview_urls in /source.json.
2020-01-21 16:34:03 -06:00
evazion
e42881fbbf Fix #4262: Exception when using Twitter video image links. 2020-01-15 15:20:33 -06:00
evazion
aff3d3b18f Fix various rubocop issues. 2020-01-11 19:01:40 -06:00
BrokenEagle
114637f594 Use more efficient use of regexes for Twitter hashtag normalization 2020-01-10 15:30:54 +00:00
BrokenEagle
2eef73e8fe Add additional Twitter hashtag
- It was mentioned on issue #4047 but was forgotten to be included
2020-01-10 14:52:49 +00:00
evazion
faeec18efc twitter: add hashtag normalization test.
Add test for #4243. Also fix warning from bootsnap:

    iseq.rb:18: warning: nested repeat operator '+' and '?' was replaced with '*' in regular expression: /(?<!\A)生誕祭(?:\d+)?\z/
2020-01-05 17:38:30 -06:00
BrokenEagle
67860a845c Add normalization for Twitter hashtags 2020-01-05 23:01:50 +00:00
evazion
309821bf73 rubocop: fix various style issues. 2019-12-22 21:23:37 -06:00
evazion
da84e3a2f2 twitter: replace twitter gem with our own API client.
The twitter gem had several problems:

* It's been unmaintained for over a year.
* It pulled in a lot of dependencies, many of which were outdated. In
  particular, it locked the `http` gem to version 3.3, preventing us
  from upgrading to 4.2.
* It raised exceptions on normal error conditions, like for deleted
  tweets or suspended users, which we really don't want.
* We had to wrap it to provide caching.

Changes:

* Fixes #4226 (Exception when creating new artists entries for suspended
  Twitter accounts)
* Drop support for scraping images from summary cards. Summary cards
  are the previews you get when you link to a website in a tweet. These
  preview images aren't always the best image.
2019-12-13 17:27:03 -06:00
evazion
a8896b664d twitter: fix batch bookmarklet selecting wrong image.
Fix regression in 7e465aeda.

https://danbooru.donmai.us/forum_topics/9127?page=276#forum_post_158779
2019-08-06 10:42:45 -05:00
evazion
7e465aedae Fix #4110: New Twitter image urls are broken in bookmarklet. 2019-08-04 20:23:10 -05:00
evazion
0f513d1a1b twitter: include intent url in new artist entries (#4028). 2018-12-27 15:03:11 -06:00
evazion
6a7cd6ce8e Fix #3984: Twitter: undefined method `first' for nil:NilClass.
Fix Sources::Strategies::Twitter#image_urls to return an empty array
instead of nil when the tweet doesn't contain any images.
2018-11-11 17:41:32 -06:00
evazion
5cf6a43918 sources: fix sources sometimes choosing wrong strategy (fix #3968)
Fix sources choosing the wrong strategy when the referer belongs to a
different site (for example, when uploading a twitter post with a pixiv
referer).

* Fix `match?` to only consider the main url, not the referer.

* Change `match?` to match against a list of domains given by the `domains` method.

* Change `match?` to an instance method.
2018-11-04 13:00:17 -06:00
evazion
f135a7c064 twitter: normalize canonical urls.
Normalize http://mobile.twitter.com to http://twitter.com in canonical urls.
2018-09-16 15:03:47 -05:00
evazion
bd47641601 twitter: don't fail when api key isn't configured. 2018-09-16 15:03:47 -05:00
evazion
325120ee51 twitter: fix parsing of the artist name from the url.
Fixes URLs like https://twitter.com/intent/user?user_id=123 being
incorrectly normalized to http://twitter.com/intent/ in artist entries.

Also fixes the artist name to be taken from the url when it can't be
obtained from the api (when the tweet is deleted).
2018-09-16 15:03:23 -05:00
evazion
583f8457f0 artists: clean up artist finding logic.
Rename Artist#find_all_by_url to url_matches and drop previous
url_matches method, along with find_artists and search_for_profile.

Previously find_artists tried to lookup the url, referer url, and profile
url in turn until an artist match was found. This was wasteful, because
the source strategy already knows which url to lookup (usually the profile
url). If that url doesn't find a match, then the artist doesn't exist.
2018-09-11 20:14:46 -05:00
Albert Yi
a5df178bcc Merge pull request #3886 from r888888888/source-api-caching
cache api clients
2018-09-11 17:34:25 -07:00
Albert Yi
4972c998f8 rely on preview urls if available for gallery 2018-09-11 15:06:12 -07:00
Albert Yi
266c7c0d5b cache api clients 2018-09-11 14:19:17 -07:00
evazion
9a980367f6 twitter: normalize artist commentaries to nfkc (#3719)
Fixes hashtags not being interpreted when the author uses a fullwidth
number sign (#, U+FF03).

ref: https://github.com/r888888888/danbooru/issues/3719#issuecomment-419535610
2018-09-10 21:45:50 -05:00
evazion
cb2d85d925 twitter: fix profile_url for twitter.com/i/web/status/:id urls.
Fix profile_url returning nil for https://twitter.com/i/web/status/943446161586733056.
2018-09-09 19:48:34 -05:00
evazion
6c94047556 Sources::Strategies::Twitter#profile_url: fix case when url is a profile url. 2018-08-29 19:29:16 -05:00
evazion
a1044dbc19 twitter: fix handling of direct image urls without a referer url. 2018-08-29 17:14:57 -05:00
evazion
bf19ea3bd1 twitter: fix typo in ASSET regex (#3850). 2018-08-29 15:09:36 -05:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
evazion
3af82de596 Partial fix for #3719: Certain commentaries not parsed correctly 2018-08-20 23:18:26 -05:00
evazion
3ef23faae2 Fix #3669: Normalize mobile.twitter.com sources.
* Normalize sources to https://twitter.com/#{artist_name}/status/#{status_id} after upload.
* Make fetch source data use screen name instead of display name for the artist name.
2018-04-27 00:36:13 -05:00
r888888888
abce4d2551 Raise error on unpermitted params.
Fail loudly if we forget to whitelist a param instead of silently
ignoring it.

misc models: convert to strong params.

artist commentaries: convert to strong params.

* Disallow changing or setting post_id to a nonexistent post.

artists: convert to strong params.

* Disallow setting `is_banned` in create/update actions. Changing it
  this way instead of with the ban/unban actions would leave the artist in
  a partially banned state.

bans: convert to strong params.

* Disallow changing the user_id after the ban has been created.

comments: convert to strong params.

favorite groups: convert to strong params.

news updates: convert to strong params.

post appeals: convert to strong params.

post flags: convert to strong params.

* Disallow users from setting the `is_deleted` / `is_resolved` flags.

ip bans: convert to strong params.

user feedbacks: convert to strong params.

* Disallow users from setting `disable_dmail_notification` when creating feedbacks.
* Disallow changing the user_id after the feedback has been created.

notes: convert to strong params.

wiki pages: convert to strong params.

* Also fix non-Builders being able to delete wiki pages.

saved searches: convert to strong params.

pools: convert to strong params.

* Disallow setting `post_count` or `is_deleted` in create/update actions.

janitor trials: convert to strong params.

post disapprovals: convert to strong params.

* Factor out quick-mod bar to shared partial.
* Fix quick-mod bar to use `Post#is_approvable?` to determine visibility
  of Approve button.

dmail filters: convert to strong params.

password resets: convert to strong params.

user name change requests: convert to strong params.

posts: convert to strong params.

users: convert to strong params.

* Disallow setting password_hash, last_logged_in_at, last_forum_read_at,
  has_mail, and dmail_filter_attributes[user_id].

* Remove initialize_default_image_size (dead code).

uploads: convert to strong params.

* Remove `initialize_status` because status already defaults to pending
  in the database.

tag aliases/implications: convert to strong params.

tags: convert to strong params.

forum posts: convert to strong params.

* Disallow changing the topic_id after creating the post.
* Disallow setting is_deleted (destroy/undelete actions should be used instead).
* Remove is_sticky / is_locked (nonexistent attributes).

forum topics: convert to strong params.

* merges https://github.com/evazion/danbooru/tree/wip-rails-5.1
* lock pg gem to 0.21 (1.0.0 is incompatible with rails 5.1.4)
* switch to factorybot and change all references

Co-authored-by: r888888888 <r888888888@gmail.com>
Co-authored-by: evazion <noizave@gmail.com>

add diffs
2018-04-06 18:09:57 -07:00
evazion
f8a5620768 Partial fix for #3514: Handle https://twitter.com/i/web/status/:id URL. 2018-01-23 23:07:21 -06:00
r888888888
5177829aff revert twitter source change 2017-12-21 10:58:28 -08:00
r888888888
b5d72ae8d8 fixes #3422 2017-12-15 17:21:33 -08:00
evazion
eeb41d2ffd twitter: get status id from referer url if not in direct url. 2017-11-16 13:29:58 -06:00
evazion
8b70e0099b twitter: fix to handle extended tweets (fix #3254, #3072). 2017-08-05 23:12:55 -05:00
r888888888
c80291c072 fix twitter:error:forbidden catch block 2017-07-21 18:12:46 -07:00
r888888888
c5a40628e6 integrate with ccs 2017-07-20 15:25:19 -07:00
evazion
128e655aef twitter: fetch hashtags for translated tags (fixes #3171). 2017-06-17 16:15:23 -05:00
evazion
5dd3151d5b twitter: convert commentary to dtext.
* Convert hashtags and mentions to dtext links.
* Replace http://t.co urls to the actual url.
* Strip the http://t.co url linking to the tweet itself.
2017-06-16 12:56:55 -05:00
evazion
9909ff20cd twitter: memoize api response. 2017-06-16 12:56:55 -05:00
evazion
b2f62482a1 Remove artist commentary handling from download rewrite strategies. 2017-05-10 18:01:51 -05:00
Albert Yi
e8336afd81 fixes #2718: Twitter artist finding should not be case sensitive 2016-10-24 12:18:44 -07:00