Commit Graph

61 Commits

Author SHA1 Message Date
evazion
8ef72d59c1 artists: allow url_matches param to take multiple urls.
Pass as an array or space-separated string:

* https://danbooru.donmai.us/artists?search[url_matches]=https://www.pixiv.net/en/users/32777+https://www.pixiv.net/en/users/3584828
* https://danbooru.donmai.us/artists?search[url_matches][]=https://www.pixiv.net/en/users/32777&search[url_matches][]=https://www.pixiv.net/en/users/3584828
2022-04-03 02:54:30 -05:00
evazion
0d480eb832 artist urls: stop using normalized_url.
Stop the last remaining uses of the `artist_urls.normalized_url` column.
It's already no longer used by the artist finder. The only remaining
uses were by API users. Those users should use the `url` column instead.
2022-04-02 23:58:01 -05:00
evazion
54cfbf84c6 pawoo: fix www.pawoo.net urls not being normalized to pawoo.net.
Fix artist URLs like https://www.pawoo.net/@01051708 not being normalized to https://pawoo.net/@01051708.
2022-03-31 02:17:51 -05:00
evazion
a272c19b98 Fix #5078: Pixiv booth upload broken.
Allow image URLs from https://booth.pximg.net to be uploaded. Fix bug
where Booth.pm URLs were incorrectly caught by the Pixiv extractor.
2022-03-30 03:25:42 -05:00
evazion
7c887f8adc artists: fix exception when adding TwitPic urls. 2022-03-20 21:56:38 -05:00
evazion
40cbc0423c sources: add Instagram profile url normalization. 2022-03-18 18:20:29 -05:00
evazion
c6e528a073 artstation: normalize https://artstation.com/artist/username/albums/all/ urls. 2022-03-18 17:10:26 -05:00
evazion
cc54b5f730 fanbox: normalize http://www.pixiv.net/fanbox/creator/3113804/post urls. 2022-03-18 17:10:26 -05:00
evazion
26d23c49d0 pawoo: normalize https://pawoo.net/users/evazion urls. 2022-03-18 17:10:26 -05:00
evazion
10dac3ee51 artists: normalize urls added to artist entries.
When a URL is added to an artist entry, normalize it to a standard form.

Artist URLs have both a `url` column and a `normalized_url` column. The
`normalized_url` is used for artist finding and the `url` is the raw URL
entered by the user. Previously only the `normalized_url` field was
normalized; now the URL entered by the user is also converted to a
normalized form.

This means that if an URL like this is added to an artist entry:

* http://www.pixiv.net/member.php?id=1234
* http://www.pixiv.net/en/users/1234
* http://www.twitter.com/DanbooruBot/
* http://mobile.twitter.com/DanbooruBot/

It will get normalized to this:

* https://www.pixiv.net/users/1234
* https://twitter.com/DanbooruBot

This fixes problems with duplicate URLs being added to artist entries
because URLs weren't normalized to a single form.
2022-03-18 02:06:50 -05:00
evazion
03d2a86ef1 artists: normalize fc2.com profile urls. 2022-03-17 19:42:57 -05:00
evazion
ded03df1ff tests: fix more broken tests. 2022-03-15 05:14:56 -05:00
evazion
cf8b8207e2 artists: change how artist urls are normalized.
Change how artist URLs are normalized in artist entries. Don't try to secretly
convert image URLs to profile URLs in artist entries. For example, if someone puts a
Pixiv image URL in an artist entry, don't secretly try to fetch the source and
convert it into a profile URL in the `normalized_url` field.

We did this because years ago, it was standard practice to put image URLs in artist
entries. Pixiv image URLs used to contain the artist's username, so we used to put
image URLs in artist entries for artist finding purposes. But Pixiv changed it so
that image URLs no longer contained the username, so we dealt with it by adding a
`normalized_url` column to artist_urls and secretly converting image URLs to profile
URLs in this field. But this is no longer necessary because now we don't normally put
image URLs in artist entries in the first place.

Now the `profile_url` method in `Source::URL` is used to normalize URLs in artist
entries. This lets us parse various profile URL formats and normalize them into a
single canonical form.

This also removes the `normalize_for_artist_finder` method from source strategies.
Instead the `profile_url` method is used for artist finding purposes. So the profile
URL returned by the source strategy needs to be the same as the URL in the artist
entry in order for artist finding to work.
2022-03-13 03:54:17 -05:00
evazion
1609059bf4 sources: factor out Source::URL::Fanbox.
Also fix it so that we grab the full image for cover URLs like this:

* Sample: https://pixiv.pximg.net/c/1620x580_90_a2_g5/fanbox/public/images/creator/1566167/cover/QqxYtuWdy4XWQx1ZLIqr4wvA.jpeg
* Full: https://pixiv.pximg.net/fanbox/public/images/creator/1566167/cover/QqxYtuWdy4XWQx1ZLIqr4wvA.jpeg
2022-02-28 06:25:06 -06:00
evazion
c5777f360e artist urls: normalize trailing slashes and missing http://.
* Remove unnecessary trailing slashes when artist URLs are saved.
* Automatically add `http://` to new artist URLs if it's missing (before
  this was an error; now it's automatically fixed).
2022-02-22 00:17:53 -06:00
evazion
60a26af6e3 rails: add 'URL' inflection.
Make it so we can write `ArtistURL` instead of `ArtistUrl`.
2022-02-22 00:17:53 -06:00
evazion
61e7d32f78 tests: fix FC2 artist normalization url test. 2020-12-13 04:10:48 -06:00
evazion
3d90414e06 Merge pull request #4487 from BrokenEagle/fix-invalid-url
Fix invalid artist URLs being allowed
2020-06-29 17:46:13 -05:00
BrokenEagle
c21af0c853 Fix invalid artist URLs being allowed
The problem was that the Addressable parser does not catch all invalid
URL cases, so some extra checks were added in.

- hostname must contain a dot

This accounts for URLs of the following type:

http://http://something.com

which has a hostname of http.

The artist URL tests were also updated with cases which test all validation
errors.
2020-05-31 06:34:06 +00:00
evazion
29740f0bee tests: fix hentai foundry artist url test. 2020-05-29 15:36:21 -05:00
evazion
3656063a6b tests: fix unit tests.
* Move old post archive tests to post version tests.

* Fix pool tests that assumed that multiple edits by the same user
  weren't merged.

* Fix references to `is_active` and `notes` on artist model.
2020-03-21 21:07:35 -05:00
evazion
4c11e339bd artists: rename is_active flag to is_deleted.
Rename is_active to is_deleted. This is for better consistency with
other models, and to reduce confusion over what "active" means for
artists. Sometimes users think active is for whether the artist is
actively producing work.
2020-03-06 14:50:21 -06:00
evazion
5a4b24f6a0 pixiv: normalize new profile urls.
ref: https://danbooru.donmai.us/forum_topics/9127?page=290#forum_post_162222
2020-01-08 17:33:55 -06:00
evazion
309821bf73 rubocop: fix various style issues. 2019-12-22 21:23:37 -06:00
Albert Yi
79dfa6c28f fix migration for circleci 2018-12-10 17:51:57 -08:00
evazion
03cc3dfa50 artists: fix editing invalid urls in artist entries (fix #3720, #3927, #3781)
Convert to an autosave association on urls. This ensures that when we
save the artist we only validate the added urls, not bad urls that we're
trying to remove, and that url validation errors are propagated up to
the artist object.

This also fixes invalid urls being saved in the artist history despite
validation failing (#3720).
2018-10-04 19:49:16 -05:00
evazion
d294514dc0 nijie: don't crash on invalid urls or deleted works (#3919). 2018-09-23 20:08:14 -05:00
evazion
b6228505aa nijie: fix page_url method.
The id in a bare image url is the member id, not the illust id.
2018-09-23 20:08:13 -05:00
Albert Yi
aea2d77c3a Merge pull request #3907 from evazion/fix-3900
#3900: Allow to search for urls associated with artists using wildcards
2018-09-17 12:03:15 -07:00
evazion
325120ee51 twitter: fix parsing of the artist name from the url.
Fixes URLs like https://twitter.com/intent/user?user_id=123 being
incorrectly normalized to http://twitter.com/intent/ in artist entries.

Also fixes the artist name to be taken from the url when it can't be
obtained from the api (when the tweet is deleted).
2018-09-16 15:03:23 -05:00
evazion
c9b3c8d217 artist urls: add tests for controller + artist url search. 2018-09-15 19:58:55 -05:00
Albert Yi
f16c3a3f40 fix nijie specs 2018-09-11 13:27:00 -07:00
evazion
b924c2bb9c nijie: fix artist url normalization. 2018-09-09 13:17:52 -05:00
evazion
aee1906761 Fix #3738: Artist URL search should be case-insensitive for domains. 2018-09-05 19:14:24 -05:00
Albert Yi
4a56f8d160 fixes #3856 for pixiv fanbox urls 2018-09-04 12:53:58 -07:00
evazion
c689a161f6 pixiv: fix failure when normalizing pixiv stacc artist urls (#3856). 2018-08-30 19:24:44 -05:00
Albert Yi
48f2a79d13 fix artist url spec and bug with nicoseiga strategy not recognizing urls 2018-08-29 17:14:36 -07:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
Albert Yi
135b97d511 additional fixes for deviantart artist search (#3771) 2018-07-27 12:31:26 -07:00
Albert Yi
fcd80b6043 Artist urls can be prepended with a '-' to mark is inactive (#3388) 2018-05-16 16:04:05 -07:00
r888888888
abce4d2551 Raise error on unpermitted params.
Fail loudly if we forget to whitelist a param instead of silently
ignoring it.

misc models: convert to strong params.

artist commentaries: convert to strong params.

* Disallow changing or setting post_id to a nonexistent post.

artists: convert to strong params.

* Disallow setting `is_banned` in create/update actions. Changing it
  this way instead of with the ban/unban actions would leave the artist in
  a partially banned state.

bans: convert to strong params.

* Disallow changing the user_id after the ban has been created.

comments: convert to strong params.

favorite groups: convert to strong params.

news updates: convert to strong params.

post appeals: convert to strong params.

post flags: convert to strong params.

* Disallow users from setting the `is_deleted` / `is_resolved` flags.

ip bans: convert to strong params.

user feedbacks: convert to strong params.

* Disallow users from setting `disable_dmail_notification` when creating feedbacks.
* Disallow changing the user_id after the feedback has been created.

notes: convert to strong params.

wiki pages: convert to strong params.

* Also fix non-Builders being able to delete wiki pages.

saved searches: convert to strong params.

pools: convert to strong params.

* Disallow setting `post_count` or `is_deleted` in create/update actions.

janitor trials: convert to strong params.

post disapprovals: convert to strong params.

* Factor out quick-mod bar to shared partial.
* Fix quick-mod bar to use `Post#is_approvable?` to determine visibility
  of Approve button.

dmail filters: convert to strong params.

password resets: convert to strong params.

user name change requests: convert to strong params.

posts: convert to strong params.

users: convert to strong params.

* Disallow setting password_hash, last_logged_in_at, last_forum_read_at,
  has_mail, and dmail_filter_attributes[user_id].

* Remove initialize_default_image_size (dead code).

uploads: convert to strong params.

* Remove `initialize_status` because status already defaults to pending
  in the database.

tag aliases/implications: convert to strong params.

tags: convert to strong params.

forum posts: convert to strong params.

* Disallow changing the topic_id after creating the post.
* Disallow setting is_deleted (destroy/undelete actions should be used instead).
* Remove is_sticky / is_locked (nonexistent attributes).

forum topics: convert to strong params.

* merges https://github.com/evazion/danbooru/tree/wip-rails-5.1
* lock pg gem to 0.21 (1.0.0 is incompatible with rails 5.1.4)
* switch to factorybot and change all references

Co-authored-by: r888888888 <r888888888@gmail.com>
Co-authored-by: evazion <noizave@gmail.com>

add diffs
2018-04-06 18:09:57 -07:00
r888888888
4a80d6c337 refactor User#validate_sock_puppet to disable for tests 2018-01-02 14:32:38 -08:00
r888888888
461ddbf017 fixes #3422 2017-12-21 11:33:23 -08:00
r888888888
502f1298a9 fix unit tests 2017-11-20 16:30:07 -08:00
evazion
eac827973b Normalize pixiv artist urls to pixiv member page (fix #3048). 2017-05-15 14:12:08 -05:00
evazion
c0e5c400a7 tests: fix tests for Rails.cache. 2017-04-16 16:48:36 -05:00
Albert Yi
0ea7d78584 remove usage of vcr cassettes; delete unused fixtures; fix some broken unit tests 2016-12-28 15:47:28 -08:00
Albert Yi
3ad639521f fixes #2805: Improve nico seiga support 2016-12-27 16:11:22 -08:00
Albert Yi
e8336afd81 fixes #2718: Twitter artist finding should not be case sensitive 2016-10-24 12:18:44 -07:00
Toks
8e17b4936a Fix artist finder for hentai foundry
#2489
2015-08-11 20:28:58 -04:00