Fix a bug where searching for `sakana~` ranked `sakana~_(meme)` beneath
random artist tags containing the word `sakana`. Now, if the search contains
punctuation, we rank exact matches first, even for small tags. Before we
ranked exact matches for small tags lower than inexact matches for large
tags. If the search contains punctuation, it's a strong signal the user
is looking for an exact match.
Fix an exception when trying to fetch source data for URLs like
https://va.media.tumblr.com/tumblr_pgohk0TjhS1u7mrsl.mp4.
For these URLs it's not possible to use the trick where we try to open
the URL as a HTML page and scrape the post id from the HTML. Instead we
get the raw video if we try to to this.
Fix usernames not being highlighted when completing @mentions.
This also changes it so the autocomplete results don't include the '@'
in front of the name.
Minor breaking change to the /autocomplete.json API. Autocomplete
results for mentions now have the type `mention` instead of `user`.
Fix /autocomplete.json returning results like this:
[
{
"table": {
"type": "tag-word",
"label": "long hair",
"value": "long_hair",
"category": 0,
"post_count": 2818211,
"antecedent": null
}
},
]
instead of like this:
[
{
"type": "tag-word",
"label": "long hair",
"value": "long_hair",
"category": 0,
"post_count": 2818211,
},
]
Also change it so that optional attributes like `antecedent` aren't
returned if they're null. This could be a minor breaking change if users
rely on these attributes always being present.
Lastly change XML results to look like this:
<autocomplete-service-results type="array">
<autocomplete-service-result>
<type>tag-word</type>
<label>long hair</label>
<value>long_hair</value>
<category type="integer">0</category>
<post-count type="integer">2801117</post-count>
</autocomplete-service-result>
</autocomplete-service-results>
instead of like this:
<objects type="array">
<object>
<type>tag</type>
<label>hair ornament</label>
<value>hair_ornament</value>
<category type="integer">0</category>
<post-count type="integer">883340</post-count>
<antecedent nil="true"/>
</object>
</objects>
When completing a static metatag, such as rating: or order:,
don't include the metatag name in the autocomplete menu.
For example, when completing `rating:g`, show `general` in the
autocomplete menu, not `rating:general`.
This makes static metatags consistent with other metatags.
Switch autocomplete to match individual words in the tag, instead of
only matching the start of the tag.
For example, "hair" matches any tag containing the word "hair", not just tags
starting with "hair". "long_hair" matches all tags containing the words "long"
and "hair", which includes "very_long_hair" and "absurdly_long_hair".
Words can be in any order and words can be left out. So "closed_eye" matches
"one_eye_closed". "asuka_langley_souryuu" matches "souryuu_asuka_langley".
This has several advantages:
* You can search characters by first name. For example, "miku" matches "hatsune_miku".
"zelda" matches both "princess_zelda" and "the_legend_of_zelda".
* You can find the right tag even if you get the word order wrong, or forget a word.
For example, "eyes_closed" matches "closed_eyes". "hair_over_eye" matches "hair_over_one_eye".
* You can find more related tags. For example, searching "skirt" shows all tags
containing the word "skirt", not just tags starting with "skirt".
The downside is this may break muscle memory by changing the autocomplete order of
some tags. This is an acceptable trade-off.
You can get the old behavior by writing a "*" at the end of the tag. For
example, searching "skirt*" gives the same results as before.
Render the HTML for autocomplete results server-side instead of in
Javascript. This is cleaner than building HTML in Javascript, but it may
hurt caching because the HTTP responses are larger.
Fixes#4698: user autocomplete contains links to /posts
Also fixes a bug where tag counts in the autocomplete menu were different
from tag counts displayed elsewhere because of differences in rounding.
Previously if you typed e.g. "/tr" in autocomplete we would first check
if "/tr" was aliased to another tag before expanding out the abbreviation.
This was for compatibility with legacy shortcut aliases. These aliases
have been removed so this is dead code now.
Also change the /autocomplete.json API to no longer strip '-' and '~'
from the start of the tag. This may be a breaking change if third-party
scripts relied on this behavior.
This was an obsolete URL format briefly used by Pixiv around 2019-2020.
There were only ~80 posts with sources using this format. They have been
manually fixed.
Bug: When uploading a direct Pixiv image URL, we ignored it in favor of the
image URL returned by the Pixiv API. This meant if you tried to upload the
original version of a revised image, we would get the revised version instead.
Fix: When given a direct Pixiv image URL, use it as-is if it's a full
image URL. If it's a sample image URL, ignore it in favor of the full image
URL as returned by the API, unless the post is deleted and the API data
is unavailable.
Foundation changed their HTML page format and we can no longer scrape
the image URL directly from the page. Instead we have to build it based
on API data.
Add ability to search the /media_assets index by AI tags. Multi-tag
searches are supported, including AND/OR/NOT operators, but metatags
aren't supported. Multi-tag searches will probably be slow.
The default AI tag confidence threshold is 50%. There's a hidden
search[min_score] URL param that lets you change this.
* Fix the suggested tags list in the related tags box not showing rating tags.
* Fix the suggested tags list showing tags that have been aliased to another tag.
Add a Suggested tags list to the Related Tags box. The suggested tags
are just the AI tags for the post.
Suggested tags are currently hidden in CSS for beta testing. Use custom
CSS to unhide them.
Exclude the posts, post_votes, favorites, media_assets, and ai_tags
tables from the BigQuery dumps. These usually take too long to complete
and also consume huge amounts of memory in the background workers.
Add a database model for storing AI-predicted tags, and add a UI for browsing and searching these tags.
AI tags are generated by the Danbooru Autotagger (https://github.com/danbooru/autotagger). See that
repo for details about the model.
The database schema is `ai_tags (media_asset_id integer, tag_id integer, score smallint)`. This is
designed to be as space-efficient as possible, since in production we have over 300 million
AI-generated tags (6 million images and 50 tags per post). This amounts to over 10GB in size, plus
indexes.
You can search for AI tags using e.g. `ai:scenery`. You can do `ai:scenery -scenery` to find posts
where the scenery tag is potentially missing, or `scenery -ai:scenery` to find posts that are
potentially mistagged (or more likely where the AI missed the tag).
You can browse AI tags at https://danbooru.donmai.us/ai_tags. On this page you can filter by
confidence level. You can also search unposted media assets by AI tag.
To generate tags, use the `autotag` script from the Autotagger repo, something like this:
docker run --rm -v ~/danbooru/public/data/360x360:/images ghcr.io/danbooru/autotagger ./autotag -c -f /images | gzip > tags.csv.gz
To import tags, use the fix script in script/fixes/. Expect a Danbooru-size dataset to take
hours to days to generate tags, then 20-30 minutes to import. Currently this all has to be done by hand.
The following metatags no longer count against the tag search limit:
* is
* id
* date
* age
* filesize
* filetype
* parent
* child
* md5
* width
* height
* duration
* mpixels
* ratio
* score
* upvotes
* downvotes
* favcount
* embedded
* tagcount
* pixiv_id
* pixiv
These are mostly metatags that have to do with properties of the post
itself. Other metatags still count because they involve things like
subqueries or joins, or they're more tag-like in function.
Add a system for upgrading accounts using upgrade codes. Users purchase
an upgrade code off-site then redeem it on-site to upgrade their account
to Gold. Upgrade codes are randomly pre-generated and are one time use
only. Codes have enough randomness that guessing a code is infeasible.
* Add "general" rating.
* Rename "safe" rating to "sensitive".
* Change safe mode to include both rating:s and rating:g.
* Treat rating:safe as a synonym for rating:sensitive.
* Link "howto:rate" in the post edit form.
This reverts commit 80ced3e418.
This turned out to be intentional. Rounding the aspect ratio to 2
decimal places is so that searches for exact ratios like `ratio:16:9` or
`ratio:1.78` work even when the ratio doesn't exactly match. Rounding to
2 decimal places means that the ratio: metatag has a 1% error tolerance.
