Commit Graph
100 Commits
Author SHA1 Message Date
evazion ffdd5e6128 posts: make post votes work the same way as comment votes.
Change post votes to work the same way as comment votes:

* Make the upvote arrow blue if you've upvoted the post, or grey if you
  haven't. Likewise for the downvote arrow.
* Make it so you can click the upvote or downvote arrows to undo the vote.
* Don't show any notices when you vote on a post.

Also fix it so that votes work the same way on the posts page, the
comments page, and in the modqueue. Before it wasn't possible to undo
votes on the comments page or in the modqueue.
2021-01-29 02:21:08 -06:00
evazion 0a8fe506b4 posts: add script to remove duplicate votes.
There were 67 duplicate votes in the production database.
2021-01-29 02:09:30 -06:00
evazion af84386d77 webpack: fix hot reloading not working.
Upgrade to webpacker-6.0.0.beta.4 to fix a bug where bin/webpack-dev-server
didn't respect the host/port config options in config/webpacker.yml,
which made it listen on the wrong port, breaking hot reloading.
2021-01-29 02:09:30 -06:00
evazion 90cd3293eb Upgrade to Webpacker 6.0. 2021-01-28 00:22:49 -06:00
evazion 2eeee446a5 js: upgrade to MobX 6.0. 2021-01-27 17:46:17 -06:00
evazion 0713751e3c tests: fix user upgrades not being tested in CI. 2021-01-27 16:55:46 -06:00
evazion 716cdb0126 tests: fix broken tests.
* Fix a broken Twitter profile image upload test.
* Skip a broken DeviantArt flash file upload test (flash no longer
  supported by DeviantArt?)
* Skip user upgrade tests when Stripe is not configured.
2021-01-27 00:40:53 -06:00
evazion 3137284292 Update ruby gems and yarn packages. 2021-01-27 00:02:35 -06:00
evazion 0919acb861 unicorn: increase workers, reduce timeout. 2021-01-26 19:52:43 -06:00
evazion 78892e6239 posts: add index on rating column.
Should improve performance for rating:e and rating:q searches. Rating:s
isn't isn't indexed because Postgres is unlikely to use the index for
rating:s searches (the selectivity is too low, ~77% of all posts are
rating:s).
2021-01-26 19:24:18 -06:00
evazion f7ea4917e5 db: sync production schema with declared schema.
Fix various minor inconsistencies between the production database schema
and the declared schema in db/structure.sql.

* tags.category was a smallint instead of an integer in production.
* The unique_schema_migrations index didn't exist outside production.
* The index_posts_on_tag_index index was called index_posts_on_tags_index
  outside production.
* The posts.tag_index column didn't have a statistics target defined
  outside production.
* ID sequences didn't have `AS integer` defined in production.
2021-01-26 19:19:20 -06:00
evazion dca53fb0ee /login: fix missing </section> tag. 2021-01-24 21:01:54 -06:00
evazion 019bcc4ed2 uploads: direct anon users to login page.
When an anonymous users tries to go to the new upload page, direct them
to the login page instead of showing them an "Access Denied" error.
Fixes complaints from SEO tools about linking to pages that return a 403
error.
2021-01-24 20:46:01 -06:00
evazion 2c06766c9e gems: workaround meta_request failure in Rails 6.1.
meta_request fails in Rails 6.1 with a `SystemStackError: stack level
too deep` error. Switch to a patched fork until the mainline gem is
fixed.

* hxxps://www.github.com/dejan/rails_panel/pull/177
* hxxps://www.github.com/dejan/rails_panel/issues/178
2021-01-24 19:26:25 -06:00
evazion 7ed4523152 Update ruby gems and yarn packages. 2021-01-24 19:18:04 -06:00
evazion ab0df8b042 Update CHANGELOG.md 2021-01-23 18:31:48 -06:00
evazion 9a9fbcc398 sessions: fix error when an anonymous user tried to logout.
Fix an exception when a user who was already logged out tried to logout
again.
2021-01-23 18:22:43 -06:00
evazion b6e06ee6fc Fix #4676: Series pool shown as blue unless hovered.
Broken in 49bc2364 by the use of @extend.

Here's what happened. There are two CSS rules that both apply to pool links:

* a:link { color: var(--link-color); }
* .pool-category-series a { color: var(--series-pool-color); }

These rules have equal specificity (0-1-1). This means the rule that is
defined last takes priority. This means the order in which CSS files are
included matters. 49bc2364 used the @extend directive in a rule for
popup menus, which required an @import, which changed the order of the
CSS files, which made the a:link rule suddenly take priority over the
series pool rule.

The proper fix would be to use Sass's new @use directive instead of
@import, but that requires the latest version of Sass, which requires
the latest version of Webpacker, which we can't upgrade to yet because
of breaking changes.

The real moral of the story is: our CSS is very fragile because of
specificity rules. It's very important that rules are defined in a
certain order, otherwise our CSS will break.

* https://sass-lang.com/documentation/at-rules/use
* https://developer.mozilla.org/en-US/docs/Web/CSS/Specificity
2021-01-23 16:01:08 -06:00
evazion b689c9cbed comments: add uniqueness constraint on votes. 2021-01-23 14:44:11 -06:00
evazion 5061b0a013 comments: change 'Shortlink' to 'Copy ID'. 2021-01-23 14:34:53 -06:00
evazion 5fd33ad545 posts: fix favorite icon.
Fix the favorite button using the outlined heart icon instead of the
solid heart icon.
2021-01-23 14:33:45 -06:00
evazion 7629a99030 Fix #4675: Comment tooltip doesn't show properly on phone.
Possible fix for #4675. This is still wonky; tapping the menu icon now
opens the menu, but doesn't close it. You have to tap outside the menu
to hide it.
2021-01-23 01:16:46 -06:00
evazion b5670898fe posts: fix mis-sized upvote icon in advanced tooltips.
Passing a class to svg_icon_tag accidentally overrode the other classes
we tried to set.
2021-01-23 00:45:13 -06:00
evazion e14bc77249 comments: rename "Post as moderator" to "Sticky" 2021-01-23 00:27:40 -06:00
evazion 3ef1e1bb60 tag relationships: remove redundant tag_matches method.
Already handled by `search_attributes`.
2021-01-22 05:22:53 -06:00
evazion 9d71ece55d comments: remove 2 comments per hour limit.
Remove the rule that Members could only post 2 bumping comments per
hour.

This was frequently misunderstood as meaning that Members could only
post 2 comments per hour. In fact, Members could post an unlimited
number of comments per hour, but the rest of their comments had to be
non-bumping. The error message we showed to users was misleading. Even
our own code misunderstood what this did when describing the config
option.

Gold users also weren't subject to this limit, which was unfair since
Gold users aren't any better at commenting than regular users. The fact
that a large number of users already ignored bump limits and nobody
really noticed indicates that the limit was unnecessary.
2021-01-22 05:16:45 -06:00
evazion 2eecf4d695 comments: let mods click score to see votes. 2021-01-22 04:26:55 -06:00
evazion bed5fdafb8 comments: add dtext shortlink option to comment menu.
Add a Shortlink menu option to the comment menu. Clicking this will copy
a DText comment shortlink (e.g. `comment #12345`) to the clipboard. You
can middle-click or right-click the menu option to get the full URL.

The menu option is called Shortlink instead of `comment #1234` because
show the full comment ID in the menu makes the menu look too unbalanced.

Note that the `navigator.clipboard` API can only be used in a https://
environment. It won't work in non-HTTPS development environments. ngrok
can help with this.
2021-01-22 03:55:48 -06:00
evazion 2beac62fc3 comments: move Javascript into component.
This is a minor breaking changing for userscripts that may have tried to
hook into our comment Javascript.
2021-01-21 20:08:14 -06:00
evazion c718667d46 comments: remove unused <a name="comment-#{id}"> element. 2021-01-21 15:41:26 -06:00
evazion 25bf78f9eb comments: fix incorrect comment links in mentions.
Fix mentions incorrectly linking to comments like this:

    https://danbooru.donmai.us/posts/1234#comment-5678

when it should have been this

    https://danbooru.donmai.us/posts/1234#comment_5678

Change comment links to this to ensure they're permanent:

    https://danbooru.donmai.us/comments/5678

The comment show action will redirect from there to the post with the
comment in context.
2021-01-21 15:36:49 -06:00
evazion 168db63807 comments: raise limit to 20 posts per page on /comments index. 2021-01-21 07:58:50 -06:00
evazion 49bc2364bd comments: tweak styling of vote buttons.
* Tweak hover colors for vote buttons.
* Tweak sizes to avoid the vote buttons shifting when the score changes.
2021-01-21 07:58:50 -06:00
evazion f4216b323f views: change upvote/downvote icons to arrows.
Change the upvote and downvote icons for posts, comments, and BURs from
thumbs-up / thumbs-down icons to up-arrow / down-arrow icons.
2021-01-21 07:58:50 -06:00
evazion 63e3b4b447 views: factor out FontAwesome icons.
Factor out FontAwesome icons into a set of helpers. This is so that it's
easier to keep track of which icons we're using and easier to change
icons globally.
2021-01-21 07:58:50 -06:00
evazion 748fcdddcf forum: fixup forum vote layout issue.
Fix problem with forum votes being laid out in a vertical list instead
of a horizontal list. Broken in 5780ed57.
2021-01-21 07:58:50 -06:00
evazion c9570e698b comments: add fix script to remove duplicate votes.
There are about 100 duplicate comment votes. This is because there
wasn't a uniqueness constraint in the database to prevent duplicate
votes. This adds a script to remove duplicate votes so that a constraint
can be added later.
2021-01-21 07:58:50 -06:00
evazion 9f313c94d1 comments: remove rule that you can't upvote your own comments.
Remove this rule for several reasons:

* A single upvote usually isn't enough to matter, especially with the
  new comment threshold.
* It felt weird that trying to vote on a comment could fail.
* Disabling the upvote button on your own comments feels weird.
* Most other sites allow you to upvote your own comments.
* You're allowed to upvote your own uploads, so it doesn't make sense
  that you can't upvote your own comments.
2021-01-21 07:58:50 -06:00
evazion 9efb374ae5 comments: allow swapping votes.
Allow users to upvote a comment, then downvote it, without raising an
error or having to manually remove the upvote first. The upvote is
automatically removed and replaced by the downvote.

Changes to the /comment_votes API:

* `POST /comment_votes` and `DELETE /comment_votes` now return a comment
  vote instead of a comment.
* The `score` param in `POST /comment_votes` is now 1 or -1, not
  `up` or `down.`
2021-01-21 07:58:50 -06:00
evazion c31f2003d9 dtext: change [quote] styling.
Change [quote] styling to Reddit/Github style (greyed out, with a bar on
the left side) instead of traditional forum style (big grey box around
the quote).

Reasons for this change:

* Box style is bulkier, especially for short quotes in comments.
* Box style works poorly for deeply nested quotes.
* Box style looks bad on widescreen monitors, since the quote stretches
  across the full width of the screen.
2021-01-20 04:41:21 -06:00
evazion 5780ed5768 comments: add scores, rework comment menu.
* Add comment scores.
* Rework voting buttons so that you can click the upvote/downvote
  buttons to toggle votes.
* Hide the edit, delete, undelete, and report buttons behind a popup menu.
* Show the upvote/downvote/reply buttons to logged out users. Redirect
  them to the login page instead.
2021-01-20 04:41:21 -06:00
evazion ccae422961 Factor out popup menu component.
Factor out the popup menu inside user tooltips into a reusable
component.
2021-01-19 20:22:41 -06:00
evazion 90567bfc61 routes: remove unused commentary index route. 2021-01-19 14:07:28 -06:00
evazion 027359a2c1 comments: fix modreport notices being visible to Members.
Fix the "This comment has been reported" notice being visible for
Members.
2021-01-19 06:05:51 -06:00
evazion 40bf74220c comments: add score to search form. 2021-01-19 05:48:41 -06:00
evazion 9af407c94b comments: set default comment threshold to -8.
* Set the default comment threshold to -8. This means that comments are
  hidden at -8 or lower and greyed out at -4 or lower.

* Reset the comment threshold to -8 for anyone with a threshold greater
  than -8. For reference, only about ~3100 users had a non-default
  threshold. About 1600 of those had their threshold reset to -8.

* Change the comment threshold to a less-than-or-equal comparison
  instead of a less-than comparsion. This means that a threshold of 0
  before is the same as a threshold of -1 now. Since everyone's
  thresholds were reset, this only affects people whose thresholds were
  already less than -8, which is so low that the difference shouldn't
  matter much.

* Set the maximum comment threshold to 5. For reference, less than 1% of
  comments have a score greater than 5.

* Set the minimum comment threshold to -100. For reference, the most
  downvoted comment has a score of -60.
2021-01-19 05:48:25 -06:00
evazion e1e3604f46 comments: rework deleted comments.
Let users see when a post has deleted comments. Show normal users a
'[deleted]' placeholder when a comment is deleted. Show the full comment
to moderators.

Also fix it so that the comment creator can't edit or undelete deleted
comments, and users can't vote on or report deleted comments.

Finally, hide the creator_id, updater_id, and body of deleted comments
in the API.
2021-01-19 04:34:51 -06:00
evazion 07bdc6eab0 comments: rework thresholded comments.
Previously thresholded comments were hidden completely. You had to click
the "Show X hidden comments" button to unhide all hidden comments in a
thread. Now it works like this:

* When a comment is below your threshold, the comment text is hidden and
  replaced by a `[hidden]` link, which you can click to unhide the comment.

* When a comment is at half your threshold (for example, your threshold
  is -8 but the comment is at -4), then the comment is greyed out.

This means that comments aren't completely hidden, they're just
collapsed, so you can see the commenter and the score without unhiding
the comment. It also means you don't have to scroll back up to unhide a
comment, and threads aren't disrupted by comments being secretly
hidden (which is confusing when people are replying to hidden comments,
which forces you to go back up and unhide to find).
2021-01-19 04:07:33 -06:00
evazion b6008b02b4 Update ruby gems and yarn packages. 2021-01-17 23:28:45 -06:00
evazion 054ac51d47 policies: remove current request from context.
This refactors Pundit policies to only rely on the current user, not on
the current user and the current HTTP request. In retrospect, it was a
bad idea to include the current request in the Pundit context. It bleeds
out everywhere and there are many contexts (in tests and models) where
we only have the current user, not the current request. The previous
commit got rid of the only two places where we used it.
2021-01-17 00:57:59 -06:00
evazion 6671711784 dmails, emails: refactor to use Rails signed_id.
Refactor email verification links and Dmail share links to use the new
Rails signed_id mechanism, rather than our own handrolled mechanism.

For Dmail share links, we have to override some Rails internal methods
so that our old links still work. For email verification links, this
will invalidate existing links, but this isn't a huge deal since these
links are short-lived anyway.

https://api.rubyonrails.org/classes/ActiveRecord/SignedId.html
https://api.rubyonrails.org/classes/ActiveRecord/SignedId/ClassMethods.html
2021-01-17 00:24:02 -06:00
evazion 6ca007ee1f Fix #4670: Replace RequestStore with AS::CurrentAttributes.
This also requires replacing CurrentUser.name with CurrentUser.user.name
because the `name` method had a conflict with CurrentAttributes.
2021-01-16 12:43:20 -06:00
evazion 6f6ec6592d Fix #4668: Add running commit to /status page. 2021-01-16 12:13:14 -06:00
evazion 32dd14f461 Remove /admin/dashboard page.
This page was just a combination of the forum listing and the bulk
update requests listing. It got zero hits in the last week.
2021-01-16 03:32:11 -06:00
evazion 0306cc16b9 modreports: add search form to /moderation_reports. 2021-01-16 02:19:29 -06:00
evazion 50a75e2ec5 modreports: allow users to see modreports they've submitted. 2021-01-16 02:19:25 -06:00
evazion e7cdae33b2 modreports: remove dead code. 2021-01-16 01:34:38 -06:00
evazion 22de059e88 docker: update Postgres container to Postgres 13.1. 2021-01-16 01:02:56 -06:00
evazion b4530183f4 Fix #4525: Show mod report notices next to reported content. 2021-01-16 01:02:42 -06:00
evazion 37792bd5dd forum posts: refactor to use ViewComponent. 2021-01-16 01:02:42 -06:00
evazion 724d87f68a comments: refactor to use ViewComponent. 2021-01-16 01:02:42 -06:00
evazion d408ccbd41 users: remove option to disable autocomplete.
This option was originally added in issue #1747. But only ~350 users
ever disabled autocomplete, only ~120 of these were seen in the last
year, and only 9 new users who signed up in the last year disabled it.

Users wishing to disable autocomplete can use this CSS:

    .ui-autocomplete { display: none !important: }

or this Javascript:

    $("[data-autocomplete]").autocomplete("disable");
2021-01-15 02:03:54 -06:00
evazion 4719a5ed1c users: set default settings in ruby instead of in database.
Specify the default settings for new users inside the User model instead
of inside the database. This makes it easier to change defaults, and it
makes the code clearer.
2021-01-15 02:03:54 -06:00
evazion 99d447279b users: remove option to disable keyboard shortcuts.
Remove the enable_post_navigation option. This option was originally
added to disable the next/prev post navbar beneath posts. It was later
repurposed to disable keyboard shortcuts.

Users who don't want keyboard shortcuts are advised to not press random
buttons on the keyboard like a caveman.

Only ~1200 users disabled this option and only ~600 were seen in the
last year.
2021-01-15 02:03:54 -06:00
evazion b6fef7f3f3 users: remove option to disable search navbar.
Remove the enable_sequential_post_navigation option. This option was
used to disable the next/previous post navbar below posts.

This option was originally added in issue #674 because of people
complaining about the navbar when it was originally added. Also there
were complaints about URLs being uglier because of search params in the
URL (e.g. /posts/1234?q=touhou). There were also various minor bugs with
it at the time, such as keyboard shortcuts not working correctly, or the
page not remembering your search after a tag edit.

These complaints are irrelevant nowadays because a) people are used to
the navbar by now (and more often complain about it *not* being there
for order:score searches), b) post URLs always contain the search now,
this option hasn't disabled that for years, and c) the initial bugs with
it were fixed years ago.

Only ~1000 users disabled this option and only ~600 were seen in the last year.

Users still wishing to hide the search navbar can use custom CSS instead.
2021-01-15 02:03:54 -06:00
evazion c48ee9d390 posts: show search navbar for all users.
Include the "q" URL param (as in `/posts/1234?q=<search>`) on thumbnails
for all users. Previously it was only for logged in users. This lets the
next/previous post navbar beneath posts be used by logged out users.
2021-01-15 02:03:54 -06:00
evazion 966a7aa71c posts: remove unused data-views data attribute. 2021-01-14 21:17:57 -06:00
evazion 1b30b71a07 posts: refactor post previews to use ViewComponent.
Refactor the post preview html to use the ViewComponent framework. This
lets us encapsulate all the HTML, CSS, and helper methods for a UI
component in a single place.

See https://viewcomponent.org.
2021-01-14 21:17:57 -06:00
evazion 097220fd88 robots.txt: allow /favicon.ico. 2021-01-13 21:26:34 -06:00
evazion 2caed533c0 Add CHANGELOG.md.
Better late than never. Old entries still need to be backfilled. The
format roughly follows the conventions of https://keepachangelog.com.
2021-01-12 19:30:20 -06:00
evazion b72d249ee3 tests: add user events controller tests. 2021-01-12 19:30:20 -06:00
evazion 7537edb211 user events: let mods only see login/logout/signup events.
Adjust permissions on user events to let Moderators only see login,
logout, and user creation events, not other types of events (password
changes, etc). Admins can see everything. These other types of events
are meant for account security purposes and aren't very relevant for
sockpuppet detection purposes.
2021-01-12 19:29:37 -06:00
evazion be5cc3f99b migrations: fixup migration error on Postgres 10.x.
`CREATE OPERATOR` only accepts the `FUNCTION` argument since Postgres 11
or higher.
2021-01-12 13:12:33 -06:00
evazion 81708d2ef8 search: log extra search metadata to NewRelic.
When a user does a tag search, log a few more things, including the normalized
search string, the number of tags in the search string, and the number of results.
2021-01-12 04:25:11 -06:00
evazion c05868e7f1 users: log restricted signups to NewRelic.
When a new user creates an account and their account is automatically
restricted, log the reason why to NewRelic.
2021-01-12 04:25:11 -06:00
evazion ceeed1e692 pagination: refactor page limits.
Refactor page limits to a) be explicitly listed in the User class (not
hidden away in the Danbooru config) and b) explicitly depend on the
CurrentUser (not implicitly by way of Danbooru.config.max_numbered_pages).
2021-01-11 21:09:06 -06:00
evazion e9464d0ea5 tests: replace xip.io with nip.io.
https://xip.io tends to go down at random times, breaking our tests. Use
https://nip.io instead.
2021-01-11 21:09:06 -06:00
evazion aab9fd710b tests: fix broken Danbooru::Http redirect tests.
Switch from https://httpbin.org to https://nghttp2.org/httpbin.
Redirects on https://httpbin.org have been broken for months, with no
resolution in sight.
2021-01-11 20:41:10 -06:00
evazion ef177a09cf searchable: fixup bugs in e7b454686. 2021-01-11 19:47:20 -06:00
evazion a698ca2b8c Fix #4663: Tag script mode sometimes adds "null" tag. 2021-01-11 19:27:10 -06:00
evazion c1b865b160 searchable: add more enum attribute search options.
Add `<enum>_not` and `<enum>_id_<op>` search options:

* https://danbooru.donmai.us/mod_actions?search[category_not]=post_regenerate,post_regenerate_iqdb
* https://danbooru.donmai.us/mod_actions?search[category_not]=48,49
* https://danbooru.donmai.us/mod_actions?search[category_id]=40..50
* https://danbooru.donmai.us/mod_actions?search[category_id_not]=40..50
* https://danbooru.donmai.us/mod_actions?search[category_id_gt]=40&search[category_id_lt]=50
2021-01-11 19:13:35 -06:00
evazion e7b454686e searchable: refactor where_operator method.
Refactor the `where_operator` method so we can use it to avoid raw SQL
in more places.
2021-01-11 19:13:29 -06:00
evazion 6d2eeb6f28 searchable: fix being unable to use multiple operators on same attribute.
Fix searches like this not working:

* https://danbooru.donmai.us/tags?search[id]=1..100&search[id_not]=50

Before one of these params would override the other.
2021-01-11 14:59:04 -06:00
evazion 1e7a5ba49d Fix ruby warnings about deprecated keyword arguments. 2021-01-11 05:12:09 -06:00
evazion 95f39afcd9 tests: fixup broken tests. 2021-01-11 05:12:09 -06:00
evazion 1878fc4049 Update ruby gems and yarn packages. 2021-01-11 05:12:09 -06:00
evazion be1251b6be autocomplete: optimize various types of bogus input.
Optimize autocomplete to ignore various types of bogus input that will
never match anything. It turns out it's not uncommon for people to do
things like paste random URLs into autocomplete, or hold down keys, or
enter long strings of gibberish text (sometimes in other languages).
Some things, like autocorrect and slash abbreviations, become
pathologically slow when fed certain types of bad input.

Autocomplete will abort and return nothing in the following situations:

* Searching for URLs (tags that start with http:// or https://).
* Overly long tags (strings longer than the 170 char tag name limit).
* Slash abbreviations longer than 10 chars (e.g. typing `/qwoijqoiqogirqewgoi`).
* Slash abbreviations that aren't alphanumeric (e.g. typing `/////////`).
* Autocorrect input that contains too much punctuation and not enough actual letters.
2021-01-11 05:12:09 -06:00
evazion fc5db679e4 autocomplete: optimize searching by artist/wiki page other names.
Optimize searches for non-English phrases in autocomplete. These
searches were pretty slow, and could sometimes cause sitewide lag spikes
when users typed long strings of non-English text into the search box
and caused an unintentional DoS.

The trick is to use an `array_to_tsvector(other_names) USING gin` index
on other_names. This supports fast string prefix matching against all
elements of the array. The downside is that it doesn't allow infix or
suffix matches, so we can't support wildcards in general. Wildcards
didn't quite work anyway, since artist and wiki other names can contain
literal '*' characters.
2021-01-10 03:35:12 -06:00
evazion d18dc573fb artists: fix misnormalization of emoji in other names.
Fix `normalize_whitespace` to not strip zero-width joiner characters
(U+200D). These characters are used in emoji and stripping them breaks
some artist other names that use emoji.
2021-01-10 02:46:20 -06:00
evazion 0899194f6b Fix conflict between normalize and array_attribute macros.
Fix the `normalize` and `array_attribute` macros conflicting with each
other on the WikiPage model. This meant code like
`wiki_page.other_names = "foo bar"` didn't work. Both macros defined a
`other_names=` method, but one method overrode the other.

The fix is to use anonymous modules and prepend so we can chain method
calls with super.
2021-01-10 02:03:12 -06:00
evazion 5962152ee3 wiki pages, artists: fix normalization of other names.
Fix wiki pages and artists to normalize other names more consistently
and correctly.

For wiki pages, we strip leading / trailing / repeated underscores to
fix user typos, and we normalize to NFKC form to make search more consistent.

For artists, we allow leading / trailing / repeated underscores because
some artist names have these, and we normalize to NFC form because some
artists have weird names that would be lost by NFKC form. This does make
search less consistent.
2021-01-10 02:03:12 -06:00
evazion e788d8d0b6 saved searches: fix normalization of labels.
Fix saved searches to remove additional invalid characters from labels:

* Remove repeated spaces or underscores.
* Remove leading and trailing spaces or underscores.
* Normalize Unicode characters to NFC form.

Also add a fix script to renormalize labels in old saved searches. A few
problems with existing searches:

* Some saved searches somehow had labels containing NULL elements.
* Some had leading or trailing underscores.
* Some had repeated underscores.
* Some had non-English characters in uppercase.
2021-01-10 02:03:12 -06:00
evazion 7fbac34962 wiki pages: fix normalization of wiki page title.
Fix the wiki page title "azur___lane" being normalized to "azur__lane"
instead of "azur_lane".
2021-01-10 02:03:12 -06:00
evazion ff6e640fcb tests: add normalize_attribute helper method.
Add a custom Shoulda matcher for testing that a model correctly normalizes an attribute.

Usage:

    subject { build(:wiki_page) }
    should normalize_attribute(:title).from(" Azur  Lane ").to("azur_lane")
2021-01-10 02:03:12 -06:00
evazion 9759701071 search: add way to search array attributes by regex.
Add a `where_any_in_array_matches_regex` method and expose it to the API:

 * https://danbooru.donmai.us/artists?search[any_other_name_matches_regex]=^blah
 * https://danbooru.donmai.us/wiki_pages?search[any_other_name_matches_regex]=^blah
 * https://danbooru.donmai.us/saved_searches?search[any_label_matches_regex]=^blah

In SQL, this does `WHERE '^blah' ~<< ANY(other_names)`, where `~<<` is a
custom operator based on the `~` regex match operator, but with the
arguments reversed. This allows it to be used with the ANY(array) operator.

See also:

* https://stackoverflow.com/a/22101172
* https://www.postgresql.org/docs/current/sql-createfunction.html
* https://www.postgresql.org/docs/current/sql-createoperator.html
* https://www.postgresql.org/docs/current/functions-comparisons.html
2021-01-10 02:03:02 -06:00
evazion 65adcd09c2 users: track logins, signups, and other user events.
Add tracking of certain important user actions. These events include:

* Logins
* Logouts
* Failed login attempts
* Account creations
* Account deletions
* Password reset requests
* Password changes
* Email address changes

This is similar to the mod actions log, except for account activity
related to a single user.

The information tracked includes the user, the event type (login,
logout, etc), the timestamp, the user's IP address, IP geolocation
information, the user's browser user agent, and the user's session ID
from their session cookie. This information is visible to mods only.

This is done with three models. The UserEvent model tracks the event
type (login, logout, password change, etc) and the user. The UserEvent
is tied to a UserSession, which contains the user's IP address and
browser metadata. Finally, the IpGeolocation model contains the
geolocation information for IPs, including the city, country, ISP, and
whether the IP is a proxy.

This tracking will be used for a few purposes:

* Letting users view their account history, to detect things like logins
  from unrecognized IPs, failed logins attempts, password changes, etc.
* Rate limiting failed login attempts.
* Detecting sockpuppet accounts using their login history.
* Detecting unauthorized account sharing.
2021-01-08 22:34:37 -06:00
evazion 94e125709c users: add Restricted user level.
Add a Restricted user level. Restricted users are level 10, below
Members. New users start out as Restricted if they sign up from a proxy
or an IP recently used by another user.

Restricted users can't update or edit any public content on the site
until they verify their email address, at which point they're promoted
to Member. Restricted users are only allowed to do personal actions
like keep favorites, keep favgroups and saved searches, mark dmails as
read or deleted, or mark forum posts as read.

The restricted state already existed before, the only change here is
that now it's an actual user level instead of a hidden state. Before it
was based on two hidden flags on the user, the `requires_verification`
flag (set when a user signs up from a proxy, etc), and the `is_verified`
flag (set after the user verifies their email). Making it a user level
means that now the Restricted status will be shown publicly.

Introducing a new level below Member means that we have to change every
`is_member?` check to `!is_anonymous` for every place where we used
`is_member?` to check that the current user is logged in.
2021-01-07 17:10:29 -06:00
evazion da3e8e4726 searchable: fix bug with searching multiple association attributes.
Fix a bug with searches like the following not working correctly:

* https://danbooru.donmai.us/comments.json?search[creator][level]=20&search[creator_id]=1234
* https://danbooru.donmai.us/comments.json?search[creator][level]=20&search[creator_name]=abcd
* https://danbooru.donmai.us/comments.json?search[post][rating]=s&search[post_tags_match]=touhou

It wasn't possible to search for both `creator` and `creator_id` at the
same time (or `post` and `post_tags_match`, etc). Only the `creator_id`
param would be recognized.

Also refactor some internals:

* `search_includes` was renamed to `search_associated_attribute`.
* `search_attribute` was split up into `search_basic_attribute` and
  `search_associated_attribute`.
2021-01-07 17:10:29 -06:00
evazion e356fd0fd5 models: remove nonexistent associations.
Remove the WikiPageVersion#artist and Post#updater associations. Neither
of these existed and they caused problems with searching includable
associations in the API.
2021-01-07 17:10:29 -06:00
evazion 59d64a76f3 tests: fix debug_mode option.
Fix debug mode to only re-raise unexpected exceptions. Fixes debug mode
breaking controller tests that expected to throw exceptions.
2021-01-07 17:10:29 -06:00