Changes:
* Change the `expires_at` field to `duration`.
* Make moderators choose from a fixed set of standard ban lengths,
instead of allowing arbitrary ban lengths.
* List `duration` in seconds in the /bans.json API.
* Dump bans to BigQuery.
Note that some old bans have a negative duration. This is because their
expiration date was before their creation date, which is because in 2013
bans were migrated to Danbooru 2 and the original ban creation dates
were lost.
* Export daily public database dumps to BigQuery and Google Cloud Storage.
* Only data visible to anonymous users is exported. Some tables have
null or missing fields because of this.
* The bans table is excluded because some bans have an expires_at
timestamp set beyond year 9999, which BigQuery doesn't support.
* The favorites table is excluded because it's too slow to dump (it
doesn't have an id index, which is needed by find_each).
* Version tables are excluded because dumping them every day is
inefficient, streaming insertions should be used instead.
Links:
* https://console.cloud.google.com/bigquery?project=danbooru1
* https://console.cloud.google.com/storage/browser/danbooru_public
* https://storage.googleapis.com/danbooru_public/data/posts.json
If we detect that the session cookie has expired (by the presence of the
`#login_illust` element on the page), then clear the cached session
cookie. The current source fetch will still fail, but the next fetch
will try to login again and hopefully succeed.
Put the option to sticky a comment in the "..." popup menu instead of
in the comment edit form. This makes it more consistent with deleting or
undeleting a comment.
Also fix a bug where the comment undelete icon didn't show up due to a
typo.
Allow users to view their own rate limits with /rate_limits.json.
Note that rate limits are only updated after every API call, so this
page only shows the state of the limits after the last call, not the
current state.
If the user makes a request while rate limited, penalize them 1 second
for that request, up to a maximum of 30 seconds. This means that if a
user doesn't stop making requests after being rate limited, then they
will stay rate limited forever until they stop.
This is to temp ban bots, especially spam bots, that flood requests
while ignoring HTTP errors or rate limits.
(Note that this is on a per-endpoint basis. Being rate limited on one
endpoint won't penalize you for making calls to other endpoints.)
* Tie rate limits to both the user's ID and their IP address.
* Make each endpoint have separate rate limits. This means that, for
example, your post edit rate limit is separate from your post vote
rate limit. Before all write actions had a shared rate limit.
* Make all write endpoints have rate limits. Before some endpoints, such
as voting, favoriting, commenting, or forum posting, weren't subject
to rate limits.
* Add stricter rate limits for some endpoints:
** 1 per 5 minutes for creating new accounts.
** 1 per minute for login attempts, changing your email address, or
for creating mod reports.
** 1 per minute for sending dmails, creating comments, creating forum
posts, or creating forum topics.
** 1 per second for voting, favoriting, or disapproving posts.
** These rate limits all have burst factors high enough that they
shouldn't affect normal, non-automated users.
* Raise the default write rate limit for Gold users from 2 per second to
4 per second, for all other actions not listed above.
* Raise the default burst factor to 200 for all other actions not listed
above. Before it was 10 for Members, 30 for Gold, and 60 for Platinum.
Rework the rate limit implementation to make it more flexible:
* Allow setting different rate limits for different actions. Before we
had a single rate limit for all write actions. Now different
controller endpoints can have different limits.
* Allow actions to be rate limited by user ID, by IP address, or both.
Before actions were only limited by user ID, which meant non-logged-in
actions like creating new accounts or attempting to login couldn't be rate
limited. Also, because actions were limited by user ID only, you could
use multiple accounts with the same IP to get around limits.
Other changes:
* Remove the API Limit field from user profile pages.
* Remove the `remaining_api_limit` field from the `/profile.json` endpoint.
* Rename the `X-Api-Limit` header to `X-Rate-Limit` and change it from a
number to a JSON object containing all the rate limit info
(including the refill rate, the burst factor, the cost of the call,
and the current limits).
* Fix a potential race condition where, if you flooded requests fast
enough, you could exceed the rate limit. This was because we checked
and updated the rate limit in two separate steps, which was racy;
simultaneous requests could pass the check before the update happened.
The new code uses some tricky SQL to check and update multiple limits
in a single statement.
* Refactor the paginator into a ViewComponent.
* Fix inconsistent spacing between paginator items.
* Fix a bug where the sequential paginator generated the wrong next /
previous page links in the <link rel="{next|prev}"> tags in the <head>.
* Always include the final page as a hidden html element, so that it can
be unhidden with custom CSS.
* Make it easier to change the pagination window.
Require the user to re-enter their password before they can view,
create, update, or delete their API keys.
This works by tracking the timestamp of the user's last password
re-entry in a `last_authenticated_at` session cookie, and redirecting
the user to a password confirmation page if they haven't re-entered
their password in the last hour.
This is modeled after Github's Sudo mode.
Track when an API key was last used, which IP address last used it, and
how many times it's been used overall.
This is so you can tell when an API key was last used, so you know if
the key is safe to delete, and so you can tell if an unrecognized IP has
used your key.
Add the ability to restrict API keys so that they can only be used with
certain IP addresses or certain API endpoints.
Restricting your key is useful to limit damage in case it gets leaked or
stolen. For example, if your key is on a remote server and it gets
hacked, or if you accidentally check-in your key to Github.
Restricting your key's API permissions is useful if a third-party app or
script wants your key, but you don't want to give full access to your
account.
If you're an app or userscript developer, and your app needs an API key
from the user, you should only request a key with the minimum
permissions needed by your app.
If you have a privileged account, and you have scripts running under
your account, you are highly encouraged to restrict your key to limit
damage in case your key gets leaked or stolen.
* Add an explanation of what an API key is and how to use it.
* Make it possible for the site owner to view all API keys.
* Remove the requirement to re-enter your password before you can view
your API key (to be reworked).
* Move the API key controller from maintenance/user/api_keys_controller.rb
to a top level controller.
Regression caused by the switch from the mobile API to the Ajax API. In
the Ajax API, commentaries have /jump.php?<url> links that we have to strip out.
* fix the test for R18 posts, which are now supported
* replace the dead post for the imageMap test with an active one
* fix the ordering of images: it was possible to get the wrong image in
some cases when not using the batch bookmarklet, because of how fanbox
orders imageMap posts
* fix a bug where the sample wasn't properly replaced with the full
size
* rename test file to match the rest of the other files
Fix the Pixiv API no longer working by rewriting the Pixiv strategy to
use the Ajax API instead of the mobile API.
Before we could authenticate in the mobile API by using the OAuth 2.0
grant_type=password authentication flow. This no longer works. Now it
requires logging in through a HTML page, which is protected by Google
reCaptcha. This makes using the mobile API infeasible.
Instead we switch to the Ajax API, which only needs a PHPSESSID to
authenticate. This can be obtained by logging in manually and using the
devtools to extract the cookie.
This also temporarily removes support for Pixiv novels. This should be
moved to a separate source strategy.
