* Have CI build Docker images for both x86 and ARM.
* Add a `bin/rails danbooru:docker:build-arm` command for building a Docker image locally for ARM.
Usage:
* Test the image:
docker run --rm -it --platform linux/arm64 ghcr.io/danbooru/danbooru bash
* Build the image:
bin/rails danbooru:docker:build-arm
* Build the image by hand:
git archive HEAD | docker buildx build - --platform linux/amd64 --build-arg SOURCE_COMMIT=$(git rev-parse HEAD) -t danbooru -f Dockerfile --load
The Danbooru image now requires at least Docker 20.10.10 to run. If you
get weird errors, check `docker version` and make sure you're running a
recent enough version of Docker.
This is because Ubuntu 22.04 uses Glibc 2.34, which uses the clone3
syscall, which was blocked by Docker's default seccomp policy up until
20.10.10 [1].
You may have to upgrade your distro or install Docker manually [2] if
your distro doesn't ship a recent enough version of Docker.
A workaround for older versions of Docker is to use the
`--security-opt seccomp=unconfined` option to disable seccomp [3].
[1] https://pascalroeleven.nl/2021/09/09/ubuntu-21-10-and-fedora-35-in-docker/
[2] https://docs.docker.com/engine/install/
[3] https://docs.docker.com/engine/security/seccomp/
When the artist name couldn't found for a Newgrounds URL, for example
for `https://www.newgrounds.com/dump/item`, then the `profile_url`
method erroneously returned `https://.newgrounds.com`. This led to an
error later on when the artist finder tried to parse the invalid URL.
Also fix `strategy_should_work` to test that the profile URL is a valid
URL, and not to try to download the file when image_urls is empty.
* Fix `AST.tag` to downcase the tag name.
* Change PostQuery::Parser to use build nodes using `AST.tag`,
`AST.metatag`, `AST.wildcard`, etc methods instead of building nodes
directly. This way all the normalization happens in the node
constructor methods instead of in the parser.
Fixes this warning:
Calling `DidYouMean::SPELL_CHECKERS.merge!(error_name => spell_checker)' has been deprecated.
Please call `DidYouMean.correct_error(error_name, spell_checker)' instead.
Don't record unknown url params that don't come from our app. This
includes typos, url params from userscripts, and weird params from
broken bots, crawlers, or other unknown sources. Indexing too many
params can lead to a mapping explosion.
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html
Don't record most HTTP request and response headers in the APM, except
for the User-Agent, Referer, Save-Data, X-Forwarded-For, Accept-Language,
and Content-Type headers. Recording every HTTP header for every request
takes up a lot of space and most of them aren't very useful.
Eliminate an unnecessary `SELECT name, post_count, category FROM tags WHERE name IN (?)`
query on the post show page. This query was used by `humanized_essential_tag_string`
when generating the page title. This query was redundant because we already use
another query to load the tag list with `SELECT * FROM tags WHERE name IN (?)`.
Raise an error if the search is invalid for one of the following reasons:
* It contains multiple conflicting order: metatags (e.g. `order:score order:favcount` or `ordfav:a ordfav:b`).
* It contains a metatag that can't be used more than once: (e.g. `limit:5 limit:10`, `random:5 random:10`).
* It contains a metatag that can't be negated (e.g. `-order:score`, `-limit:20`, or `-random:20`).
* It contains a metatag that can't be used in an OR clause (e.g. ` touhou or order:score`, `touhou or limit:20`, `touhou or random:20`).
Fix queries like `(fate_(series) saber)` being parsed as `fate_(series` + `saber)`
instead of `fate_(series)` + `saber`.
This is pretty hacky. We assume that parentheses in tags are balanced.
So the rule is that trailing parentheses are part of the tag as long as
they're balanced, and not part of the tag if they're unbalanced.
Fix queries like `(~a ~b) (~c ~d)` being handled like `~a ~b ~c ~d`.
Caused by trimming AND nodes from the tree before rewriting the '~'
operator, which caused `~a` terms to be incorrectly lifted out of
subexpressions.
Switch to the post search engine using the new PostQuery parser. The new
engine fully supports AND, OR, and NOT operators and grouping expressions
with parentheses.
Highlights:
New OR operator:
* `skirt or dress` (same as `~skirt ~dress`)
Tags can be grouped with parentheses:
* `1girl (skirt or dress)`
* `(blonde_hair blue_eyes) or (red_hair green_eyes)`
* `~(blonde_hair blue_eyes) ~(red_hair green_eyes)` (same as above)
* `(pantyhose or thighhighs) (black_legwear or brown_legwear)`
* `(~pantyhose ~thighhighs) (~black_legwear ~brown_legwear)` (same as above)
Metatags can be OR'd together:
* `user:evazion or fav:evazion`
* `~user:evazion ~fav:evazion`
Wildcard tags can combined with either AND or OR:
* `black_* white_*` (find posts with at least one black_* tag AND one white_* tag)
* `black_* or white_*` (find posts with at least one black_* tag OR one white_* tag)
* `~black_* ~white_*` (same as above)
See 4c7cfc73 for more syntax examples.
Fixes#4949: And+or search?
Fixes#5056: Wildcard searches return unexpected results when combined with OR searches
Inline the page footer, news updates, the ban notice, and the user
verification notice into the default layout. This is a micro
optimization to reduce the number of spans reported to the APM.
Don't show deprecated tags in the related tags or translated tags lists
when editing a post. It doesn't make sense to recommended adding tags
that can't be added to the post.
This is necessary for the `commentary:` metatag, which has different
behavior depending on whether the metatag value is quoted. For example,
`commentary:translated` finds translated commentaries, while
`commentary:"translated"` finds commentaries containing the literal word
"translated".
Count the number of commented and noted posts directly, instead of
indirectly by counting the number of posts in a `commenter:<name>` or
`noteupdater:<name>` search. This is faster because it generates
better SQL.
Refactor the `PostQuery::AST#simplify` method to split it into three
methods: `#trim` to eliminate redundant AND and OR clauses, `#simplify`
to expand deeply nested subexpressions, and `#sort` to sort the query
into alphabetical order.
This is so we can normalize queries written by users by parsing and
rewriting them, but without expanding out nested subexpressions, which
can substantially alter the way the query is written.
Fix artist URLs still showing old cached site icons because the URL
didn't change when the file was updated. Use `image_pack_tag` so that
the filename includes the hash, so that the URL changes when the file
changes.
Stop the last remaining uses of the `artist_urls.normalized_url` column.
It's already no longer used by the artist finder. The only remaining
uses were by API users. Those users should use the `url` column instead.
Sometimes when a Newgrounds post is part of a set, there is a list of
links to other posts in the set in the artist's commentary. Exclude
these links because they're not really part of the commentary.
Example: https://www.newgrounds.com/art/view/boxofwant/annie-hughes-1 (NSFW)
Fix searches like this:
https://danbooru.donmai.us/artist_urls?search[url_not_regex]=https://www/.artstation/.com/[a-zA-Z0-9-.]
returning a HTTP 500 error with a blank page.
Here the problem is that Postgres raises an error because `[a-zA-Z0-9-.]`
is an invalid character class (it should be `[a-ZA-Z0-9.-]`). This error
happens on the error page itself, when rendering the <link rel="next"> /
<link rel="prev"> links in the <head>, so it causes the error page to fail.
Fix mass updates of the form `mass update A -> B` not being allowed.
This was originally because after `rename` was introduced, we wanted to
prevent people from using mass updates to move tags. Now, `mass update A -> B`
adds B to all posts tagged A instead of moving A to B. So `mass update A -> B`
should no longer be disallowed.
This also makes it so that it's an error to create a mass update with a
syntax error in the search. Before searches couldn't have syntax errors,
but now with the new query parser it's possible.
Fix the Tags field in the BUR search form not finding all BURs
mentioning that tag. Specifically, tags that were part of a mass update,
and that were prefixed with `~` or `-` (OR tags and NOT tags), weren't
indexed as tags affected by the BUR.
This requires re-running script/fixes/064_initialize_bulk_update_request_tags.rb
to fix old BURs.
evazion
evazion