Commit Graph

242 Commits

Author SHA1 Message Date
evazion
3ef23faae2 Fix #3669: Normalize mobile.twitter.com sources.
* Normalize sources to https://twitter.com/#{artist_name}/status/#{status_id} after upload.
* Make fetch source data use screen name instead of display name for the artist name.
2018-04-27 00:36:13 -05:00
evazion
7d295a11d0 fixup! Fix #3654: Deviantart commentaries fail fetching. 2018-04-18 19:01:33 -05:00
evazion
c4ae2bd2fa Fix #3654: Deviantart commentaries fail fetching. 2018-04-18 18:44:56 -05:00
evazion
302994e5d9 Fix #3639: Favorite count pixiv tags aren't skipped by translated tags. 2018-04-13 22:39:52 -05:00
evazion
28cd633210 Fix #3637: Bookmarklet fails on artstation for some posts. 2018-04-13 11:18:09 -05:00
Albert Yi
37ba6714d1 fixes #3617 2018-04-12 12:11:27 -07:00
Albert Yi
d9d98f05c2 Merge branch 'rails-5.1' 2018-04-11 11:40:36 -07:00
evazion
ab25e73cd7 deviantart: replace html scraper with api client (#3260). 2018-04-09 20:53:14 -05:00
r888888888
abce4d2551 Raise error on unpermitted params.
Fail loudly if we forget to whitelist a param instead of silently
ignoring it.

misc models: convert to strong params.

artist commentaries: convert to strong params.

* Disallow changing or setting post_id to a nonexistent post.

artists: convert to strong params.

* Disallow setting `is_banned` in create/update actions. Changing it
  this way instead of with the ban/unban actions would leave the artist in
  a partially banned state.

bans: convert to strong params.

* Disallow changing the user_id after the ban has been created.

comments: convert to strong params.

favorite groups: convert to strong params.

news updates: convert to strong params.

post appeals: convert to strong params.

post flags: convert to strong params.

* Disallow users from setting the `is_deleted` / `is_resolved` flags.

ip bans: convert to strong params.

user feedbacks: convert to strong params.

* Disallow users from setting `disable_dmail_notification` when creating feedbacks.
* Disallow changing the user_id after the feedback has been created.

notes: convert to strong params.

wiki pages: convert to strong params.

* Also fix non-Builders being able to delete wiki pages.

saved searches: convert to strong params.

pools: convert to strong params.

* Disallow setting `post_count` or `is_deleted` in create/update actions.

janitor trials: convert to strong params.

post disapprovals: convert to strong params.

* Factor out quick-mod bar to shared partial.
* Fix quick-mod bar to use `Post#is_approvable?` to determine visibility
  of Approve button.

dmail filters: convert to strong params.

password resets: convert to strong params.

user name change requests: convert to strong params.

posts: convert to strong params.

users: convert to strong params.

* Disallow setting password_hash, last_logged_in_at, last_forum_read_at,
  has_mail, and dmail_filter_attributes[user_id].

* Remove initialize_default_image_size (dead code).

uploads: convert to strong params.

* Remove `initialize_status` because status already defaults to pending
  in the database.

tag aliases/implications: convert to strong params.

tags: convert to strong params.

forum posts: convert to strong params.

* Disallow changing the topic_id after creating the post.
* Disallow setting is_deleted (destroy/undelete actions should be used instead).
* Remove is_sticky / is_locked (nonexistent attributes).

forum topics: convert to strong params.

* merges https://github.com/evazion/danbooru/tree/wip-rails-5.1
* lock pg gem to 0.21 (1.0.0 is incompatible with rails 5.1.4)
* switch to factorybot and change all references

Co-authored-by: r888888888 <r888888888@gmail.com>
Co-authored-by: evazion <noizave@gmail.com>

add diffs
2018-04-06 18:09:57 -07:00
evazion
04ecca6a06 Fix #3591: DeviantArt commentaries: strip embedded image/text thumbs. 2018-03-31 11:27:14 -05:00
evazion
80219a0ccf commentaries: replace <img> tags with dtext links (#3586)
Replace <img> tags with dtext links, unless the <img> tag is already
inside an <a> tag.

Also strip Unicode spaces from DeviantArt commentaries.
2018-03-30 21:42:51 -05:00
evazion
b185efbb5f tumblr commentaries: include asker's name in ask posts (#3586). 2018-03-30 21:42:51 -05:00
evazion
3fefb73e90 Fix #3561: Tumblr: support answer posts. 2018-02-24 10:31:59 -06:00
evazion
f8a5620768 Partial fix for #3514: Handle https://twitter.com/i/web/status/:id URL. 2018-01-23 23:07:21 -06:00
r888888888
59cd4d71d3 fixes #3467 2017-12-28 14:56:03 -08:00
evazion
48b67967b5 Fix #3457: Timeout when fetching source data from NicoSeiga. 2017-12-27 13:08:14 -06:00
evazion
265377bdbb Fix #3450: Aliased tags show up under translated tags.
Resolves aliases in translated tags. For example, say we lookup `遠坂凛`
and find `tohsaka_rin` and `toosaka_rin`. We apply aliases so that
`tohsaka_rin` becomes `toosaka_rin`, which is then returned as the only
translated tag.
2017-12-23 12:27:58 -06:00
r888888888
5177829aff revert twitter source change 2017-12-21 10:58:28 -08:00
r888888888
b5d72ae8d8 fixes #3422 2017-12-15 17:21:33 -08:00
evazion
4c39783d28 Fix #3424: /iqdb_queries.json fails for certain urls.
Fix the HTML page -> image URL download rewrite strategy failing for
https://lohas.nicoseiga.jp/thumb/${id}i URLs.
2017-12-15 10:16:06 -06:00
evazion
a14c492020 Fix #3396: Pixiv novel cover IDs are parsed incorrectly. 2017-11-21 13:03:03 -06:00
evazion
f8581c8113 Fix #3321: HTML tags not converted in artist commentaries. 2017-11-21 02:42:49 -06:00
r888888888
502f1298a9 fix unit tests 2017-11-20 16:30:07 -08:00
r888888888
9d5e4f969f fix source tests 2017-11-20 12:30:29 -08:00
evazion
5837b74de4 tumblr: fix exception when fetching source data.
Fixes an `undefined method `gsub' for nil:NilClass` exception when fetching source
data for this url:

* /uploads/new?url=https%3A%2F%2Fuiti-ukumo.tumblr.com%2Fpost%2F156157443158%2F2016-12

Bug introduced in 3c524aa5: `url` wasn't initialized due to `@url = url`
being removed from Sources::Site#initialize.
2017-11-18 15:44:09 -06:00
evazion
f896c24d37 tumblr: fail more gracefully if API key is not configured. 2017-11-18 13:53:11 -06:00
evazion
22c9cfcec5 Fix #3384: Fetch commentary not creating fully formed textile links 2017-11-18 13:02:50 -06:00
r888888888
3c524aa5e6 fix bug with source::site 2017-11-17 17:26:33 -08:00
r888888888
9a3824a87e fixes #3382 2017-11-17 16:18:59 -08:00
evazion
eeb41d2ffd twitter: get status id from referer url if not in direct url. 2017-11-16 13:29:58 -06:00
evazion
cbd68ac91b /source.json: do artist lookup and return matches in artists. 2017-11-14 23:28:25 -06:00
r888888888
d4008a5ddc fixes #3327 2017-10-09 16:32:05 -07:00
r888888888
449385f08f fixes #3313 2017-10-09 12:17:15 -07:00
lllusion3469
4c3733ce5a Keep auth_secure cookie from deviantart
Unlike this projects, all other DA scrapers I found[0] require that both
"auth" and "auth_secure" are set.
While the current approach, for some reason, seems to have worked fine in
the past, it has stopped functioning on some systems, sometimes
permanently (apparently sometimes only temporarily).
I don't have the means to find out the reason behind that, since all
configurations I've tried (mobile, VPN, and regular connection +
4 seperate accounts) resulted in me being effectively logged out and
unable to view mature works after deleting the "auth_secure" cookie.
The "if auth_cookie" part might not actually be necessary, its only
purpose is to prevent unintended breakage due to lack of information
on my part.

[0]: b662b46a44 (diff-aab4e263d591d1c7f23d015a2e540a32R161)
     cbaecf5507 (diff-9f9821e7d744874a2fe42e0a69e4b14aR33)
2017-08-30 20:47:59 +02:00
r888888888
bc69badadd better error handling of failed deviant art logins 2017-08-28 16:19:41 -07:00
r888888888
d124ff8128 fix case when fetching pixiv sources when illust_id param comes before mode 2017-08-23 17:50:33 -07:00
evazion
9b031a9c20 Fix #3266: Normalize fullwidth Unicode characters in translated tags. 2017-08-12 17:13:52 -05:00
Albert Yi
521927ae35 Merge pull request #3255 from evazion/fix-3254
Fix #3254, #3072: Some Twitter URLs upload wrong image.
2017-08-07 16:03:20 -07:00
evazion
c270954991 Fix #3258: Fetch source data broken for https://i.pximg.net/* ugoiras. 2017-08-06 17:23:29 -05:00
evazion
8b70e0099b twitter: fix to handle extended tweets (fix #3254, #3072). 2017-08-05 23:12:55 -05:00
Albert Yi
36a17d79f9 Merge pull request #3242 from evazion/fix-drop-whitecube
Drop Pixiv whitecube support (#2681, #3206).
2017-07-26 17:07:27 -07:00
evazion
80e8c9d174 strategies/pixiv.rb: removed unused methods (#3206).
The page count and the moniker are now obtained from the API.
2017-07-26 18:17:04 -05:00
evazion
7ae860a8b8 Drop Pixiv whitecube support (#2681, #3206). 2017-07-26 18:06:46 -05:00
r888888888
c80291c072 fix twitter:error:forbidden catch block 2017-07-21 18:12:46 -07:00
Albert Yi
9b982b3379 Merge pull request #3228 from r888888888/ccs
integrate with ccs
2017-07-21 18:00:05 -07:00
r888888888
c5a40628e6 integrate with ccs 2017-07-20 15:25:19 -07:00
evazion
93d44c6b84 Fix #3224: 403 Forbidden when logging in to DeviantArt.
Login fails if the User-Agent isn't set while POST'ing the login
credentials.
2017-07-19 20:25:27 -05:00
r888888888
97ce9cafac add httparty option config 2017-07-12 15:52:48 -07:00
r888888888
eb6c5e3af5 switch to httparty 2017-07-12 15:52:48 -07:00
evazion
cc8986641b Fix #3208: Fix translated tag suggestions for Pixiv.
* Only suggest the Danbooru tag with the same name if there is no
  matching wiki other name. Example: if we have the Pixiv tag `Fate` and
  the Danbooru tag `fate_(series)` with other name `fate`, suggest that,
  not the Danbooru tag `fate`.

* Don't suggest tags that are empty or whose wiki is deleted.

* Only split tags on "/" if there are no other matches, and only for Pixiv.

* For Pixiv, only include traditional media tags in tag list, not digital media (Photoshop, SAI).

* Add some tests.
2017-07-09 12:10:30 -05:00