Commit Graph

26 Commits

Author SHA1 Message Date
evazion
7e471fe223 sources: replace HTTParty with Danbooru::Http in http_exists?. 2020-06-21 15:11:56 -05:00
evazion
6e6ce6e62f nijie: replace Mechanize with Danbooru::Http.
The Nijie login process works like this:

* First we submit our `email` and `password` to `https://nijie.info/login_int.php`.
* Then we save the NIJIEIEID session cookie from the response.
* We optionally retry if login failed. Nijie returns 429 errors with a
  `Retry-After: 5` header if we send too many login requests. This can
  happen during parallel testing.
* We cache the login cookies for only 1 hour so we don't have to worry
  about them becoming invalid if we cache them too long.

Cookies and retrying errors on failure are handled transparently by Danbooru::Http.
2020-06-21 05:22:57 -05:00
evazion
855e31ac90 nijie: fetch commentary as html instead of plaintext.
Fix regression in #4475. Fetch the commentary as html instead of
plaintext so that we don't lose links or other formatting.

Also fix it so that /jump.php redirect links are replaced with the
actual url.
2020-05-29 15:36:21 -05:00
evazion
2c60a51f64 Merge pull request #4475 from nonamethanks/refactor_source_normalizing
Refactor source normalization
2020-05-27 16:52:17 -05:00
nonamethanks
116f3a67ef Nijie: fetch full commentary rather than truncated preview 2020-05-22 02:47:19 +02:00
nonamethanks
307df3b3e4 Refactor source normalization
* Move the source normalization logic out of the post model
  and into individual sources' strategies.
* Rewrite normalization tests to be handled into each source's test,
  and expand them significantly. Previously we were only testing
  a very small subset of domains and variants.
* Fix up normalization for several sites.
* Normalize fav.me urls into normal deviantart urls.
2020-05-21 22:46:51 +02:00
evazion
b7350b8fe0 tests: fix various broken tests. 2020-01-17 19:21:20 -06:00
evazion
1b426fb23f Fix #4150: Nijie strategy fails for mp4 files. 2019-09-03 22:33:09 -05:00
evazion
9ecf36585c nijie: update for new image urls.
Nijie moved from this:

    https://pic03.nijie.info/nijie_picture/236014_20170620101426_0.png (page: https://www.nijie.info/view.php?id=218856)

to this:

    https://pic.nijie.net/03/nijie_picture/236014_20170620101426_0.png (page: https://www.nijie.info/view.php?id=218856)
2019-08-04 17:49:54 -05:00
evazion
bea8c2a4b8 nijie: fix failure to handle certain image urls.
Fix IMAGE_URL regex not matching urls of this form:

* https://pic04.nijie.info/nijie_picture/diff/main/287736_161475_20181112032855_1.png

This caused the illust id to not be parsed from the url, which led to `#image_url`
returning nil, which led to uploads failing because the url to download was missing.
2018-11-12 18:04:07 -06:00
evazion
8f6c710c6b tests: fix translated tags test failures. 2018-11-12 18:04:07 -06:00
evazion
d9063a9f2a nijie: support preview urls (#3919). 2018-09-24 17:08:37 -05:00
evazion
5525bbe1ca nijie: normalize all thumbnail urls (#3919). 2018-09-23 20:08:14 -05:00
evazion
d294514dc0 nijie: don't crash on invalid urls or deleted works (#3919). 2018-09-23 20:08:14 -05:00
evazion
b6228505aa nijie: fix page_url method.
The id in a bare image url is the member id, not the illust id.
2018-09-23 20:08:13 -05:00
evazion
d6235d6f9e nijie: add canonical url tests. 2018-08-31 23:23:15 -05:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
r888888888
abce4d2551 Raise error on unpermitted params.
Fail loudly if we forget to whitelist a param instead of silently
ignoring it.

misc models: convert to strong params.

artist commentaries: convert to strong params.

* Disallow changing or setting post_id to a nonexistent post.

artists: convert to strong params.

* Disallow setting `is_banned` in create/update actions. Changing it
  this way instead of with the ban/unban actions would leave the artist in
  a partially banned state.

bans: convert to strong params.

* Disallow changing the user_id after the ban has been created.

comments: convert to strong params.

favorite groups: convert to strong params.

news updates: convert to strong params.

post appeals: convert to strong params.

post flags: convert to strong params.

* Disallow users from setting the `is_deleted` / `is_resolved` flags.

ip bans: convert to strong params.

user feedbacks: convert to strong params.

* Disallow users from setting `disable_dmail_notification` when creating feedbacks.
* Disallow changing the user_id after the feedback has been created.

notes: convert to strong params.

wiki pages: convert to strong params.

* Also fix non-Builders being able to delete wiki pages.

saved searches: convert to strong params.

pools: convert to strong params.

* Disallow setting `post_count` or `is_deleted` in create/update actions.

janitor trials: convert to strong params.

post disapprovals: convert to strong params.

* Factor out quick-mod bar to shared partial.
* Fix quick-mod bar to use `Post#is_approvable?` to determine visibility
  of Approve button.

dmail filters: convert to strong params.

password resets: convert to strong params.

user name change requests: convert to strong params.

posts: convert to strong params.

users: convert to strong params.

* Disallow setting password_hash, last_logged_in_at, last_forum_read_at,
  has_mail, and dmail_filter_attributes[user_id].

* Remove initialize_default_image_size (dead code).

uploads: convert to strong params.

* Remove `initialize_status` because status already defaults to pending
  in the database.

tag aliases/implications: convert to strong params.

tags: convert to strong params.

forum posts: convert to strong params.

* Disallow changing the topic_id after creating the post.
* Disallow setting is_deleted (destroy/undelete actions should be used instead).
* Remove is_sticky / is_locked (nonexistent attributes).

forum topics: convert to strong params.

* merges https://github.com/evazion/danbooru/tree/wip-rails-5.1
* lock pg gem to 0.21 (1.0.0 is incompatible with rails 5.1.4)
* switch to factorybot and change all references

Co-authored-by: r888888888 <r888888888@gmail.com>
Co-authored-by: evazion <noizave@gmail.com>

add diffs
2018-04-06 18:09:57 -07:00
r888888888
9d5e4f969f fix source tests 2017-11-20 12:30:29 -08:00
evazion
9b031a9c20 Fix #3266: Normalize fullwidth Unicode characters in translated tags. 2017-08-12 17:13:52 -05:00
evazion
b880b07387 sources: factor out html-to-dtext code to DText.from_html. 2017-07-01 11:15:48 -05:00
evazion
294358b4a6 nijie: fetch image_urls for batch bookmarklet. 2017-06-20 16:29:31 -05:00
evazion
2d5fc191dd nijie: convert commentary to dtext. 2017-06-20 16:11:16 -05:00
evazion
25e7db860a nijie: fetch artist commentary. 2017-06-20 16:11:16 -05:00
r888888888
6174d0eef2 normalize nijie popup urls (fixes #3153) 2017-06-14 12:26:13 -07:00
r888888888
85fa58cb7c add test for #3153 2017-06-14 11:36:04 -07:00