Commit Graph

383 Commits

Author SHA1 Message Date
evazion
610391205f deviantart: fix artist finder for profile urls missing the 'www'.
Fix the artist finder to work when the profile url in the artist entry
is missing the 'www'. Example:

  https://deviantart.com/noizave
  https://www.deviantart.com/noizave
2018-09-07 11:36:48 -05:00
evazion
e37844303d deviantart: take artist name from url when unavailable from API.
In some cases we can't get the artist name from the API, either because
we can't do the API call because the url doesn't contain a deviation id,
or because the work is deleted:

* http://fc08.deviantart.net/files/f/2007/120/c/9/cool_like_me_by_47ness.jpg (work: http://fav.me/dwcohb)
* https://pre00.deviantart.net/423b/th/pre/i/2017/281/e/0/mindflayer_girl01_by_nickbeja-dbpxdt8.png (work: http://fav.me/dbpxd58)

Switch to taking the artist name from the url (when present) to deal
with these cases. Fixes the artist finder and the artist url normalizer
to work in this situation.
2018-09-03 18:27:01 -05:00
evazion
8f87fb90d9 deviantart: handle urls without deviation ids (fix #3864)
Some older URL formats don't contain the deviation id:

* http://fc08.deviantart.net/files/f/2007/120/c/9/Cool_Like_Me_by_47ness.jpg
* http://pre06.deviantart.net/8497/th/pre/f/2009/173/c/c/cc9686111dcffffffb5fcfaf0cf069fb.jpg

In these cases we can't make the API call. Fix failures due to not being
able to do API calls in this situation.

Also fix canonical_url to use the image_url when it contains the
deviation id, or the page_url when it doesn't.

Finally, fix page_url to use the url from the API instead of the raw url
given by the user, so that it's in a consistent form for canonical_url.
2018-09-03 18:26:45 -05:00
evazion
316acead16 deviantart: fix error when uploading image belonging to deleted work. 2018-09-02 23:09:40 -05:00
evazion
2d1b1311d6 deviantart: fix sample urls not being rewritten to full size urls. 2018-09-02 23:09:29 -05:00
evazion
807c3dd5f4 deviantart: remove obsolete image sample rewrite rules.
Remove rewrite rules for these types of sample urls:

* http://th00.deviantart.net/fs71/PRE/f/2014/065/3/b/goruto_by_xyelkiltrox-d797tit.png
* http://th04.deviantart.net/fs70/300W/f/2009/364/4/d/Alphes_Mimic___Rika_by_Juriesute.png

These URLs aren't served to users any more, and just stripping out "PRE"
or "200H" isn't sufficient to get the full size image. In general, an
api call is required to find the full size image url.
2018-09-02 14:49:58 -05:00
evazion
b9ed676bfb deviantart: handle origin-orig.deviantart.net urls. 2018-09-02 13:57:15 -05:00
evazion
d693f01dde Fix #3859: Related tag and find artist don't run when fetch data fails.
Fixes an exception in the artist finder caused by searching for a nil profile_url.
2018-09-01 11:48:42 -05:00
evazion
d6235d6f9e nijie: add canonical url tests. 2018-08-31 23:23:15 -05:00
evazion
736c22c3ce pixiv: fix fetch source data failing on fanbox images.
ref: https://danbooru.donmai.us/forum_posts/150000
2018-08-31 00:21:03 -05:00
evazion
a1044dbc19 twitter: fix handling of direct image urls without a referer url. 2018-08-29 17:14:57 -05:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
Albert Yi
135b97d511 additional fixes for deviantart artist search (#3771) 2018-07-27 12:31:26 -07:00
Albert Yi
a0205be8b5 fixes #3771 2018-07-06 11:44:07 -07:00
Albert Yi
5ae37597cd fixes #3728 2018-05-25 13:24:49 -07:00
Albert Yi
f3364b9892 skip earlier on failed pixiv tests 2018-05-15 16:15:59 -07:00
Albert Yi
8d5db3723b skip tests that throw an net::opentimeout error 2018-05-14 10:45:46 -07:00
Albert Yi
6839249e9d add logic for persisting pixiv sessions in tests 2018-05-10 13:59:57 -07:00
Albert Yi
5bca31bad1 add retry monkey patch for mechanize 2018-05-09 16:45:03 -07:00
Albert Yi
e23814be92 fix pixiv agent shutdown 2018-05-09 16:13:47 -07:00
Albert Yi
d4a2521eec reset connection for pixiv tests 2018-05-09 14:59:18 -07:00
Albert Yi
59dde72924 fix pixiv tests, install ffmpeg 2018-05-09 14:11:34 -07:00
Albert Yi
99012ff342 fix tests 2018-05-09 11:59:51 -07:00
evazion
4fd4cbd2a6 twitter: fix tests. 2018-04-28 12:33:05 -05:00
evazion
c4ae2bd2fa Fix #3654: Deviantart commentaries fail fetching. 2018-04-18 18:44:56 -05:00
evazion
302994e5d9 Fix #3639: Favorite count pixiv tags aren't skipped by translated tags. 2018-04-13 22:39:52 -05:00
Albert Yi
d9d98f05c2 Merge branch 'rails-5.1' 2018-04-11 11:40:36 -07:00
evazion
ab25e73cd7 deviantart: replace html scraper with api client (#3260). 2018-04-09 20:53:14 -05:00
r888888888
abce4d2551 Raise error on unpermitted params.
Fail loudly if we forget to whitelist a param instead of silently
ignoring it.

misc models: convert to strong params.

artist commentaries: convert to strong params.

* Disallow changing or setting post_id to a nonexistent post.

artists: convert to strong params.

* Disallow setting `is_banned` in create/update actions. Changing it
  this way instead of with the ban/unban actions would leave the artist in
  a partially banned state.

bans: convert to strong params.

* Disallow changing the user_id after the ban has been created.

comments: convert to strong params.

favorite groups: convert to strong params.

news updates: convert to strong params.

post appeals: convert to strong params.

post flags: convert to strong params.

* Disallow users from setting the `is_deleted` / `is_resolved` flags.

ip bans: convert to strong params.

user feedbacks: convert to strong params.

* Disallow users from setting `disable_dmail_notification` when creating feedbacks.
* Disallow changing the user_id after the feedback has been created.

notes: convert to strong params.

wiki pages: convert to strong params.

* Also fix non-Builders being able to delete wiki pages.

saved searches: convert to strong params.

pools: convert to strong params.

* Disallow setting `post_count` or `is_deleted` in create/update actions.

janitor trials: convert to strong params.

post disapprovals: convert to strong params.

* Factor out quick-mod bar to shared partial.
* Fix quick-mod bar to use `Post#is_approvable?` to determine visibility
  of Approve button.

dmail filters: convert to strong params.

password resets: convert to strong params.

user name change requests: convert to strong params.

posts: convert to strong params.

users: convert to strong params.

* Disallow setting password_hash, last_logged_in_at, last_forum_read_at,
  has_mail, and dmail_filter_attributes[user_id].

* Remove initialize_default_image_size (dead code).

uploads: convert to strong params.

* Remove `initialize_status` because status already defaults to pending
  in the database.

tag aliases/implications: convert to strong params.

tags: convert to strong params.

forum posts: convert to strong params.

* Disallow changing the topic_id after creating the post.
* Disallow setting is_deleted (destroy/undelete actions should be used instead).
* Remove is_sticky / is_locked (nonexistent attributes).

forum topics: convert to strong params.

* merges https://github.com/evazion/danbooru/tree/wip-rails-5.1
* lock pg gem to 0.21 (1.0.0 is incompatible with rails 5.1.4)
* switch to factorybot and change all references

Co-authored-by: r888888888 <r888888888@gmail.com>
Co-authored-by: evazion <noizave@gmail.com>

add diffs
2018-04-06 18:09:57 -07:00
evazion
80219a0ccf commentaries: replace <img> tags with dtext links (#3586)
Replace <img> tags with dtext links, unless the <img> tag is already
inside an <a> tag.

Also strip Unicode spaces from DeviantArt commentaries.
2018-03-30 21:42:51 -05:00
evazion
b185efbb5f tumblr commentaries: include asker's name in ask posts (#3586). 2018-03-30 21:42:51 -05:00
evazion
3fefb73e90 Fix #3561: Tumblr: support answer posts. 2018-02-24 10:31:59 -06:00
evazion
f8a5620768 Partial fix for #3514: Handle https://twitter.com/i/web/status/:id URL. 2018-01-23 23:07:21 -06:00
evazion
e47c983cfd tests: fix pawoo tags test. 2018-01-20 13:25:13 -06:00
evazion
265377bdbb Fix #3450: Aliased tags show up under translated tags.
Resolves aliases in translated tags. For example, say we lookup `遠坂凛`
and find `tohsaka_rin` and `toosaka_rin`. We apply aliases so that
`tohsaka_rin` becomes `toosaka_rin`, which is then returned as the only
translated tag.
2017-12-23 12:27:58 -06:00
r888888888
461ddbf017 fixes #3422 2017-12-21 11:33:23 -08:00
r888888888
b5d72ae8d8 fixes #3422 2017-12-15 17:21:33 -08:00
evazion
4c39783d28 Fix #3424: /iqdb_queries.json fails for certain urls.
Fix the HTML page -> image URL download rewrite strategy failing for
https://lohas.nicoseiga.jp/thumb/${id}i URLs.
2017-12-15 10:16:06 -06:00
evazion
255082d3b5 tumblr: fix test failure. 2017-11-26 15:37:51 -06:00
evazion
0b5ed163fe Fix #3395: fix pixiv API exception when uploading bad pixiv id images.
Fixes an exception when attempting to upload a Pixiv image from a
deleted work. The download strategy tries to fetch the source data in
the course of rewriting the URL, which fails if the work has been
deleted from Pixiv.

Raise a BadIDError and leave the URL as-is (don't rewrite it).
2017-11-21 14:40:39 -06:00
evazion
f8581c8113 Fix #3321: HTML tags not converted in artist commentaries. 2017-11-21 02:42:49 -06:00
r888888888
502f1298a9 fix unit tests 2017-11-20 16:30:07 -08:00
r888888888
9d5e4f969f fix source tests 2017-11-20 12:30:29 -08:00
evazion
40d0751e83 Fix NoStrategyError during artist url normalization (#3382).
Fixes a bug from 9a3824a. When an artist entry is saved, `ArtistUrl.normalize`
is called on every URL, which calls `Sources::Site.new(url)`. This
raised NoStrategyError when an artist entry contained URLs that weren't
recognized by any strategy.

This also caused `Fetch source data` to fail in certain cases when it
attempted to find the artist.
2017-11-19 10:49:30 -06:00
evazion
2422ce036c tumblr_test.rb: fix test failures. 2017-11-18 13:52:30 -06:00
evazion
22c9cfcec5 Fix #3384: Fetch commentary not creating fully formed textile links 2017-11-18 13:02:50 -06:00
evazion
6b8bc318b5 deviantart_test.rb: fix test failures. 2017-11-18 13:00:18 -06:00
Albert Yi
058783755d Merge pull request #3379 from evazion/fix-3377
Fix #3377: Batch bookmarklet doesn't fetch artist/tags from twitter
2017-11-16 12:02:06 -08:00
evazion
f633222ef0 twitter: test fetching source data from direct image with referer. 2017-11-16 13:29:58 -06:00
r888888888
cd5d9cdaeb update twitter test 2017-11-16 11:19:39 -08:00