Remove the ability to lock a tag's category. Before a moderator could
lock a tag such that only an admin could change the tag's category.
Nowadays the ability to change a tag's category is based on the tag's
size. Members can change tag categories for tags with up to 50 posts,
and Builders can change categories for tags with up to 1000 posts.
Manually locking tags is not necessary.
We only had a few dozen locked tags, mostly random *_(cosplay) tags or
company name tags. Most of these are holdovers from moderators randomly
locking tags like ten years ago.
The `is_locked` field is still in the database, so it is still returned
by the /tags.json API, even though it is unused.
Allow admins to remove votes on posts. This is for fixing vote abuse.
Votes can be removed by going to the vote list on the /post_votes page,
or by clicking on a post's score, then using the "Remove" option in the
"..." dropdown menu next to the vote.
Votes are soft-deleted - they're marked as deleted in the database, but
not fully deleted. Removed votes are only visible to admins, not to
regular users. When a vote is removed by an admin, it leaves a mod
action.
Technically it's possible to undelete votes, but there's no UI for it.
Changes:
* Make it so you can click or hover over a post's favorite count to see
the list of public favorites.
* Remove the "Show »" button next to the favorite count.
* Make the favorites list visible to all users. Before favorites were
only visible to Gold users.
* Make the /favorites page show the list of all public favorites,
instead of redirecting to the current user's favorites.
* Add /posts/:id/favorites endpoint.
* Add /users/:id/favorites endpoint.
This is for several reasons:
* To make viewing favorites work the same way as viewing upvotes.
* To make posts load faster for Gold users. Before, we loaded all the
favorites when viewing a post, even when the user didn't look at them.
This made pageloads slower for posts that had hundreds or thousands of
favorites. Now we only load the favlist if the user hovers over the favcount.
* To make the favorite list visible to all users. Before, it wasn't
visible to non-Gold users, because of the performance issue listed above.
* To make it more obvious that favorites are public by default. Before,
since regular users could only see the favcount, they may have
mistakenly believed other users couldn't see their favorites.
Make private favgroups a Gold-only option. This is for consistency with
private favorites and upvotes being Gold-only options.
Existing Members with private favgroups are allowed to keep them, as
long as they don't disable privacy. If they disable it, then they can't
re-enable it again without upgrading to Gold first.
Make private favorites and upvotes a Gold-only account option.
Existing Members with private favorites enabled are allowed to keep it
enabled, as long as they don't disable it. If they disable it, then they
can't re-enable it again without upgrading to Gold first.
This is a Gold-only option to prevent uploaders from creating multiple
accounts to upvote their own posts. If private upvotes were allowed for
Members, then it would be too easy to use fake accounts and private
upvotes to upvote your own posts.
* Allow Member-level users to vote.
* Don't allow Banned or Restricted users to create favorites any more.
Banned and Restricted users aren't allowed to upvote or favorite any
more to prevent sockpuppet accounts from upvoting even after they're
banned.
Make upvotes public the same way favorites are public:
* Rename the "Private favorites" account setting to "Private favorites and upvotes".
* Make upvotes public, unless the user has private upvotes enabled. Note
that private upvotes are still visible to admins. Downvotes are still
hidden to everyone except for admins.
* Make https://danbooru.donmai.us/post_votes visible to all users. This
page shows all public upvotes. Private upvotes and downvotes are only
visible on the page to admins and to the voter themselves.
* Make votes searchable with the `upvote:username` and `downvote:username`
metatags. These already existed before, but they were only usable by
admins and by people searching for their own votes.
Upvotes are public to discourage users from upvoting with multiple
accounts. Upvote abuse is obvious to everyone when upvotes are public.
The other reason is to make upvotes consistent with favorites, which are
already public.
Stop updating the fav_string attribute on posts. The column still exists
on the table, but is no longer used or updated.
Like the pool_string in 7d503f08, the fav_string was used in the past to
facilitate `fav:X` searches. Posts had a hidden fav_string column that
contained a list of every user who favorited the post. These were
treated like fake hidden tags on the post so that a search for `fav:X`
was treated like a tag search.
The fav_string attribute has been unused for search purposes for a while
now. It was only kept because of technicalities that required
departitioning the favorites table first (340e1008e) before it could be
removed. Basically, removing favorites with `@favorite.destroy` was
slow because Rails always deletes object by ID, but we didn't have an
index on favorites.id, and we couldn't easily add one until the
favorites table was departitioned.
Fixes#4652. See https://github.com/danbooru/danbooru/issues/4652#issuecomment-754993802
for more discussion of issues caused by the fav_string (in short: write
amplification, post table bloat, and favorite inconsistency problems).
Let all users have unlimited favorites. Formerly the limit was 10k
favorites for regular members, 20k for Gold, and unlimited for Platinum.
Limiting favorites doesn't make sense since upvotes are unlimited.
Remove the ability for users to lock ratings, note, and post statuses.
Historically the majority of locked posts were from 10+ years ago when
certain users habitually locked ratings and notes on every post they
touched for no reason. Nowadays most posts have been unlocked. Only a
handful of locked posts are left, none of which deserve to be locked.
The is_rating_locked, is_note_locked, and is_status_locked columns still
exist in the database, but aren't used.
Add a model for storing image and video metadata for uploaded files.
Metadata is extracted using ExifTool. You will need to install ExifTool
after this commit. ExifTool 12.22 is the minimum required version
because we use the `--binary` option, which was added in this release.
The MediaMetadata model is separate from the MediaAsset model because
some files contain tons of metadata, and most of it is non-essential.
The MediaAsset model represents an uploaded file and contains essential
metadata, like the file's size and type, while the MediaMetadata model
represents all the other non-essential metadata associated with a file.
Metadata is stored as a JSON column in the database.
ExifTool returns all the file's metadata, not just the EXIF metadata.
EXIF is one of several types of image metadata, hence why we call
it MediaMetadata instead of EXIFMetadata.
A MediaAsset represents an image or video file uploaded to Danbooru. It
stores the metadata associated with the image or video. This is to work
on decoupling files from posts so that images can be uploaded separately
from posts.
Allow admins to remove comment votes by other users. This is done by
clicking the comment score to get to the comment vote list, then
clicking the Remove button on every vote.
Changes:
* Change the `expires_at` field to `duration`.
* Make moderators choose from a fixed set of standard ban lengths,
instead of allowing arbitrary ban lengths.
* List `duration` in seconds in the /bans.json API.
* Dump bans to BigQuery.
Note that some old bans have a negative duration. This is because their
expiration date was before their creation date, which is because in 2013
bans were migrated to Danbooru 2 and the original ban creation dates
were lost.
Remove the `category_name` field from the `/wiki_page.json` API. This
field was originally added only because it was needed by our autocomplete
Javascript. It was also misnamed, it wasn't the tag's category name, it
was the category's ID.
Users should use `https://danbooru.donmai.us/wiki_pages.json?only=title,tag`
instead if they need this.
This triggered a N+1 query pattern when dumping wiki pages to BigQuery,
which made dumping wiki pages very slow. It also meant this field was
included in the database dump, even though it wasn't a real database
column.
Allow users to view their own rate limits with /rate_limits.json.
Note that rate limits are only updated after every API call, so this
page only shows the state of the limits after the last call, not the
current state.
* Tie rate limits to both the user's ID and their IP address.
* Make each endpoint have separate rate limits. This means that, for
example, your post edit rate limit is separate from your post vote
rate limit. Before all write actions had a shared rate limit.
* Make all write endpoints have rate limits. Before some endpoints, such
as voting, favoriting, commenting, or forum posting, weren't subject
to rate limits.
* Add stricter rate limits for some endpoints:
** 1 per 5 minutes for creating new accounts.
** 1 per minute for login attempts, changing your email address, or
for creating mod reports.
** 1 per minute for sending dmails, creating comments, creating forum
posts, or creating forum topics.
** 1 per second for voting, favoriting, or disapproving posts.
** These rate limits all have burst factors high enough that they
shouldn't affect normal, non-automated users.
* Raise the default write rate limit for Gold users from 2 per second to
4 per second, for all other actions not listed above.
* Raise the default burst factor to 200 for all other actions not listed
above. Before it was 10 for Members, 30 for Gold, and 60 for Platinum.
Rework the rate limit implementation to make it more flexible:
* Allow setting different rate limits for different actions. Before we
had a single rate limit for all write actions. Now different
controller endpoints can have different limits.
* Allow actions to be rate limited by user ID, by IP address, or both.
Before actions were only limited by user ID, which meant non-logged-in
actions like creating new accounts or attempting to login couldn't be rate
limited. Also, because actions were limited by user ID only, you could
use multiple accounts with the same IP to get around limits.
Other changes:
* Remove the API Limit field from user profile pages.
* Remove the `remaining_api_limit` field from the `/profile.json` endpoint.
* Rename the `X-Api-Limit` header to `X-Rate-Limit` and change it from a
number to a JSON object containing all the rate limit info
(including the refill rate, the burst factor, the cost of the call,
and the current limits).
* Fix a potential race condition where, if you flooded requests fast
enough, you could exceed the rate limit. This was because we checked
and updated the rate limit in two separate steps, which was racy;
simultaneous requests could pass the check before the update happened.
The new code uses some tricky SQL to check and update multiple limits
in a single statement.
Add the ability to restrict API keys so that they can only be used with
certain IP addresses or certain API endpoints.
Restricting your key is useful to limit damage in case it gets leaked or
stolen. For example, if your key is on a remote server and it gets
hacked, or if you accidentally check-in your key to Github.
Restricting your key's API permissions is useful if a third-party app or
script wants your key, but you don't want to give full access to your
account.
If you're an app or userscript developer, and your app needs an API key
from the user, you should only request a key with the minimum
permissions needed by your app.
If you have a privileged account, and you have scripts running under
your account, you are highly encouraged to restrict your key to limit
damage in case your key gets leaked or stolen.
* Add an explanation of what an API key is and how to use it.
* Make it possible for the site owner to view all API keys.
* Remove the requirement to re-enter your password before you can view
your API key (to be reworked).
* Move the API key controller from maintenance/user/api_keys_controller.rb
to a top level controller.
Like 9efb374ae, allow users to toggle between upvoting and downvoting a
post without raising an error or having to manually remove the vote
first. If you upvote a post, then downvote it, the upvote is
automatically removed and replaced by the downvote.
Other changes:
* Tagging a post with `upvote:self` or `downvote:self` is now silently
ignored when the user doesn't have permission to vote, instead of
raising an error.
* Undoing a vote that doesn't exist now does nothing instead of
returning an error. This can happen if you open the same post in two
tabs, undo the vote in tab 1, then try to undo the vote again in tab 2.
Changes to the /post_votes API:
* `POST /post_votes` and `DELETE /post_votes` now return a post vote
instead of a post.
* The `score` param in `POST /post_votes` is now 1 or -1, not `up` or
`down`.
Remove the rule that Members could only post 2 bumping comments per
hour.
This was frequently misunderstood as meaning that Members could only
post 2 comments per hour. In fact, Members could post an unlimited
number of comments per hour, but the rest of their comments had to be
non-bumping. The error message we showed to users was misleading. Even
our own code misunderstood what this did when describing the config
option.
Gold users also weren't subject to this limit, which was unfair since
Gold users aren't any better at commenting than regular users. The fact
that a large number of users already ignored bump limits and nobody
really noticed indicates that the limit was unnecessary.
Allow users to upvote a comment, then downvote it, without raising an
error or having to manually remove the upvote first. The upvote is
automatically removed and replaced by the downvote.
Changes to the /comment_votes API:
* `POST /comment_votes` and `DELETE /comment_votes` now return a comment
vote instead of a comment.
* The `score` param in `POST /comment_votes` is now 1 or -1, not
`up` or `down.`
* Add comment scores.
* Rework voting buttons so that you can click the upvote/downvote
buttons to toggle votes.
* Hide the edit, delete, undelete, and report buttons behind a popup menu.
* Show the upvote/downvote/reply buttons to logged out users. Redirect
them to the login page instead.
Let users see when a post has deleted comments. Show normal users a
'[deleted]' placeholder when a comment is deleted. Show the full comment
to moderators.
Also fix it so that the comment creator can't edit or undelete deleted
comments, and users can't vote on or report deleted comments.
Finally, hide the creator_id, updater_id, and body of deleted comments
in the API.
This refactors Pundit policies to only rely on the current user, not on
the current user and the current HTTP request. In retrospect, it was a
bad idea to include the current request in the Pundit context. It bleeds
out everywhere and there are many contexts (in tests and models) where
we only have the current user, not the current request. The previous
commit got rid of the only two places where we used it.
This option was originally added in issue #1747. But only ~350 users
ever disabled autocomplete, only ~120 of these were seen in the last
year, and only 9 new users who signed up in the last year disabled it.
Users wishing to disable autocomplete can use this CSS:
.ui-autocomplete { display: none !important: }
or this Javascript:
$("[data-autocomplete]").autocomplete("disable");
Remove the enable_post_navigation option. This option was originally
added to disable the next/prev post navbar beneath posts. It was later
repurposed to disable keyboard shortcuts.
Users who don't want keyboard shortcuts are advised to not press random
buttons on the keyboard like a caveman.
Only ~1200 users disabled this option and only ~600 were seen in the
last year.
Remove the enable_sequential_post_navigation option. This option was
used to disable the next/previous post navbar below posts.
This option was originally added in issue #674 because of people
complaining about the navbar when it was originally added. Also there
were complaints about URLs being uglier because of search params in the
URL (e.g. /posts/1234?q=touhou). There were also various minor bugs with
it at the time, such as keyboard shortcuts not working correctly, or the
page not remembering your search after a tag edit.
These complaints are irrelevant nowadays because a) people are used to
the navbar by now (and more often complain about it *not* being there
for order:score searches), b) post URLs always contain the search now,
this option hasn't disabled that for years, and c) the initial bugs with
it were fixed years ago.
Only ~1000 users disabled this option and only ~600 were seen in the last year.
Users still wishing to hide the search navbar can use custom CSS instead.
Adjust permissions on user events to let Moderators only see login,
logout, and user creation events, not other types of events (password
changes, etc). Admins can see everything. These other types of events
are meant for account security purposes and aren't very relevant for
sockpuppet detection purposes.
Add tracking of certain important user actions. These events include:
* Logins
* Logouts
* Failed login attempts
* Account creations
* Account deletions
* Password reset requests
* Password changes
* Email address changes
This is similar to the mod actions log, except for account activity
related to a single user.
The information tracked includes the user, the event type (login,
logout, etc), the timestamp, the user's IP address, IP geolocation
information, the user's browser user agent, and the user's session ID
from their session cookie. This information is visible to mods only.
This is done with three models. The UserEvent model tracks the event
type (login, logout, password change, etc) and the user. The UserEvent
is tied to a UserSession, which contains the user's IP address and
browser metadata. Finally, the IpGeolocation model contains the
geolocation information for IPs, including the city, country, ISP, and
whether the IP is a proxy.
This tracking will be used for a few purposes:
* Letting users view their account history, to detect things like logins
from unrecognized IPs, failed logins attempts, password changes, etc.
* Rate limiting failed login attempts.
* Detecting sockpuppet accounts using their login history.
* Detecting unauthorized account sharing.
Add a Restricted user level. Restricted users are level 10, below
Members. New users start out as Restricted if they sign up from a proxy
or an IP recently used by another user.
Restricted users can't update or edit any public content on the site
until they verify their email address, at which point they're promoted
to Member. Restricted users are only allowed to do personal actions
like keep favorites, keep favgroups and saved searches, mark dmails as
read or deleted, or mark forum posts as read.
The restricted state already existed before, the only change here is
that now it's an actual user level instead of a hidden state. Before it
was based on two hidden flags on the user, the `requires_verification`
flag (set when a user signs up from a proxy, etc), and the `is_verified`
flag (set after the user verifies their email). Making it a user level
means that now the Restricted status will be shown publicly.
Introducing a new level below Member means that we have to change every
`is_member?` check to `!is_anonymous` for every place where we used
`is_member?` to check that the current user is logged in.
* Remove the PostRegeneration model. Instead just use a mod action
to log when a post is regenerated.
* Change it so that IQDB is also updated when the image samples are
regenerated. This is necessary because when the images samples are
regenerated, the thumbnail may change, which means IQDB needs to be
updated too. This can happen when regenerating old images with
transparent backgrounds where the transparency was flattened to black
instead of white in the thumbnail.
* Only display one "Regenerate image" option in the post sidebar, to
regenerate both the images and IQDB. Regenerating IQDB only can be
done through the API. Having two options in the sidebar is too much
clutter, and it's too confusing for Mods who don't know the difference
between an IQDB-only regeneration and a full image regeneration.
* Add a confirm prompt to the "Regenerate image" link.
* Remove the data-is-favorited attribute from post thumbnails.
* Remove the is_favorited attribute from the /posts.json API.
* Remove the fav_string attribute from the /posts.json API (only visible
to moderators).
* Change `Post#favorited_by?` to not use the fav_string.
Further addresses #4652 by eliminating the last places where fav_string
was used.
