Commit Graph

70 Commits

Author SHA1 Message Date
BrokenEagle
158a4aa916 Fix Pixiv user profile URL to use the latest format
This will only affect new artist and commentary records going forward.
2020-06-17 07:07:33 +00:00
BrokenEagle
05f9b78ee3 Distinctly separate and label explicit/guro content in Pixiv test
This helps discern why these tests might be failing and serve as a
reminder to set the permissions for the Pixiv account correctly.
2020-06-17 07:07:33 +00:00
evazion
19727ab5c4 Merge pull request #4505 from nonamethanks/pixiv_digital_tags
Pixiv: don't blacklist digital tools anymore
2020-06-15 20:56:56 -05:00
nonamethanks
e83d07ea7b Pixiv: don't blacklist digital tools anymore 2020-06-12 04:15:20 +02:00
evazion
2d05004bef tests: don't cache pixiv sessions. 2020-06-11 00:47:12 -05:00
evazion
d6b266514b tests: disable known broken pixiv fanbox tests. 2020-06-10 18:21:44 -05:00
nonamethanks
307df3b3e4 Refactor source normalization
* Move the source normalization logic out of the post model
  and into individual sources' strategies.
* Rewrite normalization tests to be handled into each source's test,
  and expand them significantly. Previously we were only testing
  a very small subset of domains and variants.
* Fix up normalization for several sites.
* Normalize fav.me urls into normal deviantart urls.
2020-05-21 22:46:51 +02:00
evazion
49a3538933 pixiv: add support for techorus urls. 2020-03-04 00:00:39 -06:00
evazion
09046783ac pixiv: fix tests. 2020-03-03 23:54:03 -06:00
evazion
1244e02fe2 pixiv: handle new https://i-f.pximg.net urls. 2020-02-18 19:22:57 -06:00
evazion
309821bf73 rubocop: fix various style issues. 2019-12-22 21:23:37 -06:00
evazion
03d9b3feca pixiv: support new https://www.pixiv.net/artworks/:id urls. 2019-09-24 03:33:21 -05:00
evazion
8cadef2dd7 pixiv: fix illust id parsing (fix #4043).
* Tighten up illust id parsing to avoid misparsing ids from
  non-illust urls (sketch urls and novel urls).

* Move id parsing tests from post_test.rb to sources/pixiv_test.rb.

* Drop support for touch.pixiv.net urls. These urls are no longer used
  by Pixiv and aren't present as the source of any posts on Danbooru.
2019-01-13 14:28:51 -06:00
evazion
04d5b16da7 pixiv: fix failure to upload bad pixiv id images (fix #4031)
Bug: Uploading bad pixiv id images failed because the pixiv strategy
raised a BadIDError exception when the upload service checked for the
ugoira frame data.
2019-01-03 18:01:20 -06:00
evazion
2129e60b2b pixiv: include stacc url in new artist entries (#4028). 2018-12-27 15:03:11 -06:00
evazion
8f6c710c6b tests: fix translated tags test failures. 2018-11-12 18:04:07 -06:00
evazion
fbd5f6b7f2 pixiv: fix preview_urls for ugoiras (#3891). 2018-09-12 00:43:10 -05:00
evazion
37fc215d75 pixiv: fix preview_urls to use correct url (#3891). 2018-09-11 23:55:46 -05:00
Albert Yi
a5df178bcc Merge pull request #3886 from r888888888/source-api-caching
cache api clients
2018-09-11 17:34:25 -07:00
Albert Yi
4972c998f8 rely on preview urls if available for gallery 2018-09-11 15:06:12 -07:00
Albert Yi
266c7c0d5b cache api clients 2018-09-11 14:19:17 -07:00
evazion
736c22c3ce pixiv: fix fetch source data failing on fanbox images.
ref: https://danbooru.donmai.us/forum_posts/150000
2018-08-31 00:21:03 -05:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
Albert Yi
5ae37597cd fixes #3728 2018-05-25 13:24:49 -07:00
Albert Yi
f3364b9892 skip earlier on failed pixiv tests 2018-05-15 16:15:59 -07:00
Albert Yi
8d5db3723b skip tests that throw an net::opentimeout error 2018-05-14 10:45:46 -07:00
Albert Yi
6839249e9d add logic for persisting pixiv sessions in tests 2018-05-10 13:59:57 -07:00
Albert Yi
5bca31bad1 add retry monkey patch for mechanize 2018-05-09 16:45:03 -07:00
Albert Yi
e23814be92 fix pixiv agent shutdown 2018-05-09 16:13:47 -07:00
Albert Yi
d4a2521eec reset connection for pixiv tests 2018-05-09 14:59:18 -07:00
Albert Yi
59dde72924 fix pixiv tests, install ffmpeg 2018-05-09 14:11:34 -07:00
evazion
302994e5d9 Fix #3639: Favorite count pixiv tags aren't skipped by translated tags. 2018-04-13 22:39:52 -05:00
r888888888
abce4d2551 Raise error on unpermitted params.
Fail loudly if we forget to whitelist a param instead of silently
ignoring it.

misc models: convert to strong params.

artist commentaries: convert to strong params.

* Disallow changing or setting post_id to a nonexistent post.

artists: convert to strong params.

* Disallow setting `is_banned` in create/update actions. Changing it
  this way instead of with the ban/unban actions would leave the artist in
  a partially banned state.

bans: convert to strong params.

* Disallow changing the user_id after the ban has been created.

comments: convert to strong params.

favorite groups: convert to strong params.

news updates: convert to strong params.

post appeals: convert to strong params.

post flags: convert to strong params.

* Disallow users from setting the `is_deleted` / `is_resolved` flags.

ip bans: convert to strong params.

user feedbacks: convert to strong params.

* Disallow users from setting `disable_dmail_notification` when creating feedbacks.
* Disallow changing the user_id after the feedback has been created.

notes: convert to strong params.

wiki pages: convert to strong params.

* Also fix non-Builders being able to delete wiki pages.

saved searches: convert to strong params.

pools: convert to strong params.

* Disallow setting `post_count` or `is_deleted` in create/update actions.

janitor trials: convert to strong params.

post disapprovals: convert to strong params.

* Factor out quick-mod bar to shared partial.
* Fix quick-mod bar to use `Post#is_approvable?` to determine visibility
  of Approve button.

dmail filters: convert to strong params.

password resets: convert to strong params.

user name change requests: convert to strong params.

posts: convert to strong params.

users: convert to strong params.

* Disallow setting password_hash, last_logged_in_at, last_forum_read_at,
  has_mail, and dmail_filter_attributes[user_id].

* Remove initialize_default_image_size (dead code).

uploads: convert to strong params.

* Remove `initialize_status` because status already defaults to pending
  in the database.

tag aliases/implications: convert to strong params.

tags: convert to strong params.

forum posts: convert to strong params.

* Disallow changing the topic_id after creating the post.
* Disallow setting is_deleted (destroy/undelete actions should be used instead).
* Remove is_sticky / is_locked (nonexistent attributes).

forum topics: convert to strong params.

* merges https://github.com/evazion/danbooru/tree/wip-rails-5.1
* lock pg gem to 0.21 (1.0.0 is incompatible with rails 5.1.4)
* switch to factorybot and change all references

Co-authored-by: r888888888 <r888888888@gmail.com>
Co-authored-by: evazion <noizave@gmail.com>

add diffs
2018-04-06 18:09:57 -07:00
evazion
265377bdbb Fix #3450: Aliased tags show up under translated tags.
Resolves aliases in translated tags. For example, say we lookup `遠坂凛`
and find `tohsaka_rin` and `toosaka_rin`. We apply aliases so that
`tohsaka_rin` becomes `toosaka_rin`, which is then returned as the only
translated tag.
2017-12-23 12:27:58 -06:00
evazion
0b5ed163fe Fix #3395: fix pixiv API exception when uploading bad pixiv id images.
Fixes an exception when attempting to upload a Pixiv image from a
deleted work. The download strategy tries to fetch the source data in
the course of rewriting the URL, which fails if the work has been
deleted from Pixiv.

Raise a BadIDError and leave the URL as-is (don't rewrite it).
2017-11-21 14:40:39 -06:00
evazion
f8581c8113 Fix #3321: HTML tags not converted in artist commentaries. 2017-11-21 02:42:49 -06:00
r888888888
9d5e4f969f fix source tests 2017-11-20 12:30:29 -08:00
r888888888
d124ff8128 fix case when fetching pixiv sources when illust_id param comes before mode 2017-08-23 17:50:33 -07:00
evazion
c270954991 Fix #3258: Fetch source data broken for https://i.pximg.net/* ugoiras. 2017-08-06 17:23:29 -05:00
evazion
7ae860a8b8 Drop Pixiv whitecube support (#2681, #3206). 2017-07-26 18:06:46 -05:00
evazion
22980c845a tests: fix pixiv ugoira urls. 2017-07-25 00:36:18 -05:00
evazion
cc8986641b Fix #3208: Fix translated tag suggestions for Pixiv.
* Only suggest the Danbooru tag with the same name if there is no
  matching wiki other name. Example: if we have the Pixiv tag `Fate` and
  the Danbooru tag `fate_(series)` with other name `fate`, suggest that,
  not the Danbooru tag `fate`.

* Don't suggest tags that are empty or whose wiki is deleted.

* Only split tags on "/" if there are no other matches, and only for Pixiv.

* For Pixiv, only include traditional media tags in tag list, not digital media (Photoshop, SAI).

* Add some tests.
2017-07-09 12:10:30 -05:00
evazion
94e48aa6f3 pixiv: convert commentary to dtext.
Convert pixiv links to dtext in this manner:

* https://www.pixiv.net/member_illust.php?mode=medium&illust_id=1234
  => pixiv #1234 "»":[/posts?tags=pixiv:1234]

* https://www.pixiv.net/member.php?id=1234
  => "user/1234":[https://www.pixiv.net/member.php?id=1234] "»":[/artists?search[url_matches]=https://www.pixiv.net/member.php?id=1234]
2017-06-16 18:42:37 -05:00
evazion
fed23c0baa fetch source data: fix gallery warning for non-pixiv galleries. 2017-06-06 23:03:07 -05:00
r888888888
bffa1f3dc3 fix unit tests 2017-05-30 17:37:42 -07:00
evazion
757a7da49b fixup! /source.json: don't encode '&' as '&amp;' in commentaries. 2017-05-10 21:03:29 -05:00
evazion
b68cb174f5 tests: fix pixiv tests to use https://i.pixiv.net. 2017-04-15 23:14:41 -05:00
r888888888
0b8d4105aa fix tests 2017-04-04 12:39:17 -07:00
Albert Yi
0ea7d78584 remove usage of vcr cassettes; delete unused fixtures; fix some broken unit tests 2016-12-28 15:47:28 -08:00
Albert Yi
a862c5bf03 fixes #2746: Uploading from touch.pixiv.net broken 2016-10-31 12:22:32 -07:00