Add a database model for storing AI-predicted tags, and add a UI for browsing and searching these tags.
AI tags are generated by the Danbooru Autotagger (https://github.com/danbooru/autotagger). See that
repo for details about the model.
The database schema is `ai_tags (media_asset_id integer, tag_id integer, score smallint)`. This is
designed to be as space-efficient as possible, since in production we have over 300 million
AI-generated tags (6 million images and 50 tags per post). This amounts to over 10GB in size, plus
indexes.
You can search for AI tags using e.g. `ai:scenery`. You can do `ai:scenery -scenery` to find posts
where the scenery tag is potentially missing, or `scenery -ai:scenery` to find posts that are
potentially mistagged (or more likely where the AI missed the tag).
You can browse AI tags at https://danbooru.donmai.us/ai_tags. On this page you can filter by
confidence level. You can also search unposted media assets by AI tag.
To generate tags, use the `autotag` script from the Autotagger repo, something like this:
docker run --rm -v ~/danbooru/public/data/360x360:/images ghcr.io/danbooru/autotagger ./autotag -c -f /images | gzip > tags.csv.gz
To import tags, use the fix script in script/fixes/. Expect a Danbooru-size dataset to take
hours to days to generate tags, then 20-30 minutes to import. Currently this all has to be done by hand.
This is so admins can overrule flags and always have the final say in whether a
post is approved, even in the event of coordinated or sockpuppet flagging.
Fixes#4980: Way to mark flags as invalid for admins
Add a system for upgrading accounts using upgrade codes. Users purchase
an upgrade code off-site then redeem it on-site to upgrade their account
to Gold. Upgrade codes are randomly pre-generated and are one time use
only. Codes have enough randomness that guessing a code is infeasible.
* Add "general" rating.
* Rename "safe" rating to "sensitive".
* Change safe mode to include both rating:s and rating:g.
* Treat rating:safe as a synonym for rating:sensitive.
* Link "howto:rate" in the post edit form.
Refactor ratings to not be hardcoded in various places. Make it so
all ratings are defined in Post::RATINGS.
Also make it so that you can search multiple ratings at once with `rating:q,e`.
Sometimes uploads fail with this error:
Failed to replace upload_media_assets because one or more of the new
records could not be saved.
Change it so that media assets are saved individually, so that if saving
any of them fails we get a better error message.
* Remove the default list of blocked tags in safe mode.
* Change it so that tags that are blocked in safe mode are filtered out
at the database level rather than at the html level.
* Fix it so that if upgrades are disabled, only the payment button is disabled instead of the whole page.
* Fix it so that disabling upgrades with DANBOORU_USER_UPGRADES_ENABLED="false" works.
* Rename the stripe_id column to transaction_id.
* Add a new payment_processor column to identity the processor used for
this transaction (and hence, which backend system the transaction_id is for).
Factor out the Stripe code from the UserUpgrade class. Introduce a new
PaymentTransaction abstract class that represents a payment with some
payment processor, and a PaymentTransaction::Stripe class that
implements transactions with Stripe.
Note that we can't completely eliminate Stripe even though we no longer
accept payments with it because we still need to be able to look up old
payments in Stripe.
`file_key` is a random 9-character base-62 string that will be used as
the image filename in the future.
`is_public` is whether the image can be viewed without authentication or not.
Users running downstream boorus must run `bin/rails db:migrate` and
`script/fixes/109_generate_media_asset_file_keys.rb` after this commit.
The second bit of the `bit_flags` field was previously used for the
`has_cropped` flag, which is still set on many posts, so it's not safe
to reuse it for the `is_taken_down flag.
Bug: If a tag edit failed because it contained a metatag that raised an
exception, then a new post version would be created even though the edit
didn't go through. This could happen if the newpool:, fav:, favgroup:,
disapproved:, status:active, or status:banned metatags failed (for
example, because of a privilege error).
Fix: Silently ignore all errors raised when applying metatags. This way
the edit will always succeed, so erroneous post versions won't be created.
Automatically add the bad_link tag when the source is an image url from
a known site, but it can't be converted to a page url (for example, a
Twitter or Tumblr direct image link).
Automatically add the bad_source tag when the source is from a known
site, but it's not an image or page url (for example, a Twitter or Pixiv
profile url)
Don't allow the same post to be added to the same pool twice.
This was only legitimately needed in a handful of cases. It was much
more common for posts to be mistakenly added to the same pool twice.
This setting automatically added the `-status:deleted` metatag to all searches. This meant deleted
posts were filtered out at the database level, rather than at the html level. This way searches
wouldn't have less-than-full pages.
The cost was that searches were slower, mainly because post counts weren't cached. Normally when you
search for a tag, we can get the post count from the tags table. If the search is actually like
`touhou -status:deleted`, then we don't know the count and we have to calculate it on demand.
This option is being removed because it did the opposite of what people thought it did. People
thought it made deleted posts visible, when actually it made them more hidden.
Don't allow users to request aliases, implications, or renames for invalid tag names.
As a side effect, it's no longer possible to request shortcut aliases like
`/hr -> hakurei_reimu` (slash abbreviations still exist, but they can't
be overridden with aliases). Tests involving these types of aliases are
removed.
Don't treat *_(cosplay) tags as being invisibly aliased when a character tag is aliased. For example,
if toosaka_rin is aliased to tohsaka_rin, and toosaka_rin_(cosplay) is later added to a post, don't
magically translate the tag to tohsaka_rin_(cosplay). Instead, treat it as an error to create a
*_(cosplay) tag for an aliased character tag.
This removes some of the complexity and magic behavior surrounding *_(cosplay) tags.
Factor out most of the tag edit logic from the Post class to a new
PostEdit class. The PostEdit class contains the logic for parsing tags
and metatags from the tag edit string, and for determining which tags
were added or removed by the edit.
Fixes various bugs caused by not calculating the set of added or removed
tags correctly, for example when tag category prefixes were used (e.g.
`copy:touhou`) or when the same tag was added and removed in the same
edit (e.g. `touhou -touhou`).
Fixes#5123: Tag categorization prefixes bypass deprecation check
Fixes#5126: Negating a deprecated tag will still cause the warning to show
Fixes#3477: Remove tag validator triggering on tag category changes
Fixes#4848: newpool: metatag doesn't parse correctly
Disable the ability to upgrade to Platinum. The Platinum level still
exists, but users can no longer upgrade to it. Sales of Platinum are
being disabled in preparation of increasing the tag limit for Gold
users.
