Optimize autocomplete to ignore various types of bogus input that will
never match anything. It turns out it's not uncommon for people to do
things like paste random URLs into autocomplete, or hold down keys, or
enter long strings of gibberish text (sometimes in other languages).
Some things, like autocorrect and slash abbreviations, become
pathologically slow when fed certain types of bad input.
Autocomplete will abort and return nothing in the following situations:
* Searching for URLs (tags that start with http:// or https://).
* Overly long tags (strings longer than the 170 char tag name limit).
* Slash abbreviations longer than 10 chars (e.g. typing `/qwoijqoiqogirqewgoi`).
* Slash abbreviations that aren't alphanumeric (e.g. typing `/////////`).
* Autocorrect input that contains too much punctuation and not enough actual letters.
Optimize searches for non-English phrases in autocomplete. These
searches were pretty slow, and could sometimes cause sitewide lag spikes
when users typed long strings of non-English text into the search box
and caused an unintentional DoS.
The trick is to use an `array_to_tsvector(other_names) USING gin` index
on other_names. This supports fast string prefix matching against all
elements of the array. The downside is that it doesn't allow infix or
suffix matches, so we can't support wildcards in general. Wildcards
didn't quite work anyway, since artist and wiki other names can contain
literal '*' characters.
Fix the `normalize` and `array_attribute` macros conflicting with each
other on the WikiPage model. This meant code like
`wiki_page.other_names = "foo bar"` didn't work. Both macros defined a
`other_names=` method, but one method overrode the other.
The fix is to use anonymous modules and prepend so we can chain method
calls with super.
Add tracking of certain important user actions. These events include:
* Logins
* Logouts
* Failed login attempts
* Account creations
* Account deletions
* Password reset requests
* Password changes
* Email address changes
This is similar to the mod actions log, except for account activity
related to a single user.
The information tracked includes the user, the event type (login,
logout, etc), the timestamp, the user's IP address, IP geolocation
information, the user's browser user agent, and the user's session ID
from their session cookie. This information is visible to mods only.
This is done with three models. The UserEvent model tracks the event
type (login, logout, password change, etc) and the user. The UserEvent
is tied to a UserSession, which contains the user's IP address and
browser metadata. Finally, the IpGeolocation model contains the
geolocation information for IPs, including the city, country, ISP, and
whether the IP is a proxy.
This tracking will be used for a few purposes:
* Letting users view their account history, to detect things like logins
from unrecognized IPs, failed logins attempts, password changes, etc.
* Rate limiting failed login attempts.
* Detecting sockpuppet accounts using their login history.
* Detecting unauthorized account sharing.
Add a Restricted user level. Restricted users are level 10, below
Members. New users start out as Restricted if they sign up from a proxy
or an IP recently used by another user.
Restricted users can't update or edit any public content on the site
until they verify their email address, at which point they're promoted
to Member. Restricted users are only allowed to do personal actions
like keep favorites, keep favgroups and saved searches, mark dmails as
read or deleted, or mark forum posts as read.
The restricted state already existed before, the only change here is
that now it's an actual user level instead of a hidden state. Before it
was based on two hidden flags on the user, the `requires_verification`
flag (set when a user signs up from a proxy, etc), and the `is_verified`
flag (set after the user verifies their email). Making it a user level
means that now the Restricted status will be shown publicly.
Introducing a new level below Member means that we have to change every
`is_member?` check to `!is_anonymous` for every place where we used
`is_member?` to check that the current user is logged in.
Disable the browser's native spellchecking ability on all form inputs,
except for DText inputs. We do this by setting `spellcheck="false"` on
the <body> tag, and `spellcheck="true"` on DText <input> tags.
This fixes browsers displaying a red wavy underline beneath tags in the
tag search box, among other places. We disable spellchecking globally
because most form inputs, except for DText inputs, aren't meant for
natural English language.
Refactor fav:<name> and ordfav:<name> searches to use the favorites
table instead of the posts.fav_string.
This may be slower for fav:<name> searches. The fav_string effectively
treats favorites like secret tags on the post, so fav:<name> searches
were effectively the same as tag searches. Now they do a subquery on the
favorites table, which may not perform as well for things like multiple
fav:<name> metatags or negated fav:<name> metatags.
For ordfav:<name> searches, this may be faster. ordfav: searches had a
tag match clause (`tag_index @@ 'fav:123'`) in addition to a join on the
favs table. This was redundant, and in some cases it inhibited the query
planner from choosing a more optimal plan.
Partially addresses #4652 by eliminating another place where we depended
on the fav_string.
Log the Referer header, as well as the Sec-Fetch-* headers. These are
only sent by recent versions of Chrome; see https://www.w3.org/TR/fetch-metadata.
Require new accounts to verify their email address if any of the
following conditions are true:
* Their IP is a proxy.
* Their IP is under a partial IP ban.
* They're creating a new account while logged in to another account.
* Somebody recently created an account from the same IP in the last week.
Changes from before:
* Allow logged in users to view the signup page and create new accounts.
Creating a new account while logged in to your old account is now
allowed, but it requires email verification. This is a honeypot.
* Creating multiple accounts from the same IP is now allowed, but they
require email verification. Previously the same IP check was only for
the last day (now it's the last week), and only for an exact IP match
(now it's a subnet match, /24 for IPv4 or /64 for IPv6).
* New account verification is disabled for private IPs (e.g. 127.0.0.1,
192.168.0.1), to make development or running personal boorus easier
(fixes#4618).
Tag.category_for looked up a tag's category in the Redis cache. This was
only used in a few places (in related tags, and on the popular/missed
search pages). Get rid of this method so we can work towards getting rid
of caching tag categories in Redis.
* Refactor to move upgrade logic from UserPromotion to UserUpgrade.
* Send the recipient and the purchaser of a gifted upgrade separate
dmail notifications.
Add a model to store the status of user upgrades.
* Store the upgrade purchaser and the upgrade receiver (these are
different for a gifted upgrade, the same for a self upgrade).
* Store the upgrade type: gold, platinum, or gold-to-platinum upgrades.
* Store the upgrade status:
** pending: User is still on the Stripe checkout page, no payment
received yet.
** processing: User has completed checkout, but the checkout status in
Stripe is still 'unpaid'.
** complete: We've received notification from Stripe that the payment
has gone through and the user has been upgraded.
* Store the Stripe checkout ID, to cross-reference the upgrade record on
Danbooru with the checkout record on Stripe.
This is the upgrade flow:
* When the user clicks the upgrade button on the upgrade page, we call
POST /user_upgrades and create a pending UserUpgrade.
* We redirect the user to the checkout page on Stripe.
* When the user completes checkout on Stripe, Stripe sends us a webhook
notification at POST /webhooks/receive.
* When we receive the webhook, we check the payment status, and if it's
paid we mark the UserUpgrade as complete and upgrade the user.
* After Stripe sees that we have successfully processed the webhook,
they redirect the user to the /user_upgrades/:id page, where we show
the user their upgrade receipt.
This upgrades from the legacy version of Stripe's checkout system to the
new version:
> The legacy version of Checkout presented customers with a modal dialog
> that collected card information, and returned a token or a source to
> your website. In contrast, the new version of Checkout is a smart
> payment page hosted by Stripe that creates payments or subscriptions. It
> supports Apple Pay, Dynamic 3D Secure, and many other features.
Basic overview of the new system:
* We send the user to a checkout page on Stripe.
* Stripe collects payment and sends us a webhook notification when the
order is complete.
* We receive the webhook notification and upgrade the user.
Docs:
* https://stripe.com/docs/payments/checkout
* https://stripe.com/docs/payments/checkout/migration#client-products
* https://stripe.com/docs/payments/handling-payment-events
* https://stripe.com/docs/payments/checkout/fulfill-orders
* Introduce an abstraction for normalizing attributes. Very loosely
modeled after https://github.com/fnando/normalize_attributes.
* Normalize wiki bodies to Unicode NFC form.
* Normalize Unicode space characters in wiki bodies (strip zero width
spaces, normalize line endings to CRLF, normalize Unicode spaces to
ASCII spaces).
* Trim spaces from the start and end of wiki page bodies. This may cause
wiki page diffs to show spaces being removed even when the user didn't
explicitly remove the spaces themselves.
The `source` command is a bash-ism and doesn't work in a strictly POSIX
shell like dash, which is the /bin/sh on Debian/Ubuntu. Use `.` instead.
https://en.wikipedia.org/wiki/Dot_(command)
