Set sensible defaults for connecting to the database. By default, we try
to connect to the `danbooru2` database running on localhost as the
`danbooru` user. These are the defaults recommended by the install
guide.
If you need to change the database settings, set DATABASE_URL in
.env.local or on the command line:
# .env.local
DATABASE_URL=postgresql://danbooru:password@localhost/danbooru2
# command line
$ DATABASE_URL=postgresql://danbooru:password@localhost/danbooru2 bin/rails server
This eliminates the need to copy script/install/database.yml.templ to
config/database.yml during installation and during deployment. This is
so that Danbooru works out of the box without extra configuration. In
particular, this is so that we can run Danbooru in a Docker container
without having to set DATABASE_URL.
* Don't set the inviter field for newly promoted users, or for Gold/Plat
upgrades.
* Clear the inviter field for paid Gold/Plat upgrades, and for users who
have a feedback or a modaction listing who invited them. This leaves
about 600 remaining users with an inviter field with no other record
of who invited them.
See #4750.
There are about 100 duplicate comment votes. This is because there
wasn't a uniqueness constraint in the database to prevent duplicate
votes. This adds a script to remove duplicate votes so that a constraint
can be added later.
* Set the default comment threshold to -8. This means that comments are
hidden at -8 or lower and greyed out at -4 or lower.
* Reset the comment threshold to -8 for anyone with a threshold greater
than -8. For reference, only about ~3100 users had a non-default
threshold. About 1600 of those had their threshold reset to -8.
* Change the comment threshold to a less-than-or-equal comparison
instead of a less-than comparsion. This means that a threshold of 0
before is the same as a threshold of -1 now. Since everyone's
thresholds were reset, this only affects people whose thresholds were
already less than -8, which is so low that the difference shouldn't
matter much.
* Set the maximum comment threshold to 5. For reference, less than 1% of
comments have a score greater than 5.
* Set the minimum comment threshold to -100. For reference, the most
downvoted comment has a score of -60.
Fix wiki pages and artists to normalize other names more consistently
and correctly.
For wiki pages, we strip leading / trailing / repeated underscores to
fix user typos, and we normalize to NFKC form to make search more consistent.
For artists, we allow leading / trailing / repeated underscores because
some artist names have these, and we normalize to NFC form because some
artists have weird names that would be lost by NFKC form. This does make
search less consistent.
Fix saved searches to remove additional invalid characters from labels:
* Remove repeated spaces or underscores.
* Remove leading and trailing spaces or underscores.
* Normalize Unicode characters to NFC form.
Also add a fix script to renormalize labels in old saved searches. A few
problems with existing searches:
* Some saved searches somehow had labels containing NULL elements.
* Some had leading or trailing underscores.
* Some had repeated underscores.
* Some had non-English characters in uppercase.
Add a Restricted user level. Restricted users are level 10, below
Members. New users start out as Restricted if they sign up from a proxy
or an IP recently used by another user.
Restricted users can't update or edit any public content on the site
until they verify their email address, at which point they're promoted
to Member. Restricted users are only allowed to do personal actions
like keep favorites, keep favgroups and saved searches, mark dmails as
read or deleted, or mark forum posts as read.
The restricted state already existed before, the only change here is
that now it's an actual user level instead of a hidden state. Before it
was based on two hidden flags on the user, the `requires_verification`
flag (set when a user signs up from a proxy, etc), and the `is_verified`
flag (set after the user verifies their email). Making it a user level
means that now the Restricted status will be shown publicly.
Introducing a new level below Member means that we have to change every
`is_member?` check to `!is_anonymous` for every place where we used
`is_member?` to check that the current user is logged in.
Fixes#1862. Very old posts with transparent backgrounds used black
instead of white backgrounds in thumbnails. This was changed in #603
(see also #1239), but the thumbnails were never regenerated.
Fix script to delete all invalid email addresses. In production there
were ~4000 users with invalid email addresses because we used to not do
any validation of emails during signup.
We used to not validate user email addresses, which means we have a lot
of users with invalid emails. This script deletes all emails that are
missing both an `@` and a `.` This amounts to about 3000 invalid emails.
There are an additional ~1000 emails that are missing just the `@` sign.
Many of these are simple typos, for example skipping the `@` or typing a
2 instead. Some of these may be manually fixable.
This fixes an issue where upgrading to Gold could fail if you had an
invalid email address, because we prefilled the buyer's email address on
the Stripe checkout page and an invalid email would cause Stripe to
throw an error.
* Include appealed posts in the modqueue.
* Add `status` field to appeals. Appeals start out as `pending`, then
become `rejected` if the post isn't approved within three days. If the
post is approved, the appeal's status becomes `succeeded`.
* Add `status` field to flags. Flags start out as `pending` then become
`rejected` if the post is approved within three days. If the post
isn't approved, the flag's status becomes `succeeded`.
* Leave behind a "Unapproved in three days" dummy flag when an appeal
goes unapproved, just like when a pending post is unapproved.
* Only allow deleted posts to be appealed. Don't allow flagged posts to be appealed.
* Add `status:appealed` metatag. `status:appealed` is separate from `status:pending`.
* Include appealed posts in `status:modqueue`. Search `status:modqueue order:modqueue`
to view the modqueue as a normal search.
* Retroactively set old flags and appeals as succeeded or rejected. This
may not be correct for posts that were appealed or flagged multiple
times. This is difficult to set correctly because we don't have
approval records for old posts, so we can't tell the actual outcome of
old flags and appeals.
* Deprecate the `is_resolved` field on post flags. A resolved flag is a
flag that isn't pending.
* Known bug: appealed posts have a black border instead of a blue
border. Checking whether a post has been appealed would require either
an extra query on the posts/index page, or an is_appealed flag on
posts, neither of which are very desirable.
* Known bug: you can't use `status:appealed` in blacklists, for the same
reason as above.
Replace the mocked services in scripts/mocked_services with Rails-level
mocked services.
The scripts in scripts/mocked_services were a set of stub Sinatra
servers used to mock the Reportbooru, Recommender, and IQDBs services
during development. They return fake data so you can test pages that use
these services.
Implementing these services in Rails makes it easier to run them. It
also lets us drop a dependency on Sinatra and drop a use of HTTParty.
To use these services, set the following configuration in danbooru_local_config.rb
or .env.local:
* reportbooru_server: http://localhost:3000/mock/reportbooru
* recommender_server: http://localhost:3000/mock/recommender
* iqdbs_server: http://localhost:3000/mock/iqdb
where `http://localhost:300` is the url for your local Danbooru server
(may need to be changed depending on your configuration).
The name AliasAndImplicationImporter is a holdover from the time before
bulk update requests existed. This was a bad name because it doesn't do
any actual importing, instead it's used for parsing and executing bulk
update requests.
Several fixes for the "This tag is under discussion" notice on the post
index page:
* Fix the notice appearing for BURs that aren't pending.
* Fix the notice never going away because of the cache never expiring.
* List all topics when a tag is involved in multiple BURs.
* Link to the forum post instead of the forum topic (fix#4421).
* Optimization: don't check for BURs when the search isn't a simple
single tag search.
* Add a `tags` field to the bulk update requests table for tracking all
tags involved in the request (excluding tags in mass updates that are
negated/optional/wildcards). Known issue: doesn't handle tag type
prefixes in mass updates correctly (e.g. `mass update foo -> artist:bar`
doesn't detect the tag `bar`).
* Allow searching the /bulk_update_requests page by tags.
We don't really need to cache the notice here, but we do it anyway to
reduce queries on the post index page.
* Move emails from users table to email_addresses table.
* Validate that addresses are formatted correctly and are unique across
users. Existing invalid emails are grandfathered in.
* Add is_verified flag (the address has been confirmed by the user).
* Add is_deliverable flag (an undeliverable address is an address that bounces).
* Normalize addresses to prevent registering multiple accounts with the
same email address (using tricks like Gmail's plus addressing).
* Rename 'privacy mode' to 'private favorites'.
* Make the private favorites setting only hide favorites, not favgroups
and not the user's uploads on their profile page.
* Make the favgroup is_public flag default to true instead of false and
fix existing favgroups to be public if the user didn't have privacy mode
enabled before.
* List _all_ public favgroups on the /favorite_groups index, not just
favgroups belonging to the current user.
* Add a /users/<id>/favorite_groups endpoint.
Fix favgroups containing post ids that don't exist. Most of these are from
typos or from old expunged posts that weren't properly removed from favgroups.
https://danbooru.donmai.us/forum_topics/9127?page=283#forum_post_160508
There was a recent outage that was caused by the read replica
(yukinoshita.donmai.us) being temporarily unavailable. The pg driver in
rails got hardstuck trying to connect to the replica, which brought down
the whole site. The app servers stopped responding and could only be
brought down with SIGKILL. Even try to boot the rails console didn't
work.
We only really used this to calculate tag counts inside Post.fast_count,
which wasn't really beneficial since the read replica is slower than the
main database.
Upgrade procedure:
$ echo 2.6.3 > .ruby-version
# upgrade ruby-build to know about the latest ruby versions.
# see also: github.com/rkh/rbenv-update
$ cd ~/.rbenv/plugins/ruby-build && git pull
# install the version of ruby specified in .ruby-version.
# see also: github.com/capistrano/rbenv/issues/83
$ rbenv install --skip-existing
# update the shell to use the latest version of ruby by default.
$ rbenv global 2.6.3
# update rubygems and bundler
$ gem update --system
# update the lockfile to use the latest version of bundler
$ bundle update --bundler
