Commit Graph

11457 Commits

Author SHA1 Message Date
evazion
80da1791df docker: upgrade base image to Ubuntu 22.04.
The Danbooru image now requires at least Docker 20.10.10 to run. If you
get weird errors, check `docker version` and make sure you're running a
recent enough version of Docker.

This is because Ubuntu 22.04 uses Glibc 2.34, which uses the clone3
syscall, which was blocked by Docker's default seccomp policy up until
20.10.10 [1].

You may have to upgrade your distro or install Docker manually [2] if
your distro doesn't ship a recent enough version of Docker.

A workaround for older versions of Docker is to use the
`--security-opt seccomp=unconfined` option to disable seccomp [3].

[1] https://pascalroeleven.nl/2021/09/09/ubuntu-21-10-and-fedora-35-in-docker/
[2] https://docs.docker.com/engine/install/
[3] https://docs.docker.com/engine/security/seccomp/
2022-04-23 18:10:00 -05:00
evazion
76d9e86724 Fix #5140: Unexpected error: PublicSuffix::DomainInvalid for searching some newgrounds urls in /artists
When the artist name couldn't found for a Newgrounds URL, for example
for `https://www.newgrounds.com/dump/item`, then the `profile_url`
method erroneously returned `https://.newgrounds.com`. This led to an
error later on when the artist finder tried to parse the invalid URL.

Also fix `strategy_should_work` to test that the profile URL is a valid
URL, and not to try to download the file when image_urls is empty.
2022-04-22 23:16:41 -05:00
evazion
db6bb2ccac Fix #5136: Regular tags are now case-sensitive.
* Fix `AST.tag` to downcase the tag name.
* Change PostQuery::Parser to use build nodes using `AST.tag`,
  `AST.metatag`, `AST.wildcard`, etc methods instead of building nodes
  directly. This way all the normalization happens in the node
  constructor methods instead of in the parser.
2022-04-22 02:14:07 -05:00
evazion
90182148aa Merge pull request #5137 from nonamethanks/foundation-videos
Foundation: fix some video posts not being extracted
2022-04-22 01:50:26 -05:00
evazion
57a92ad336 Fix #5072: Fandom source normalization is wrong 2022-04-22 01:27:17 -05:00
evazion
40dda8a672 Merge pull request #5138 from nonamethanks/fix-fandom-links
Fix normalization for fandom sources
2022-04-22 00:36:11 -05:00
evazion
5f70cf1eab Merge pull request #5139 from nonamethanks/no-empty-else
Sources: do not use an empty else in case blocks
2022-04-22 00:35:40 -05:00
evazion
2ae2c23eef Upgrade Bundler to 2.3.12.
Fixes this warning:

    Calling `DidYouMean::SPELL_CHECKERS.merge!(error_name => spell_checker)' has been deprecated.
    Please call `DidYouMean.correct_error(error_name, spell_checker)' instead.
2022-04-21 21:43:17 -05:00
evazion
7701fe2e17 Upgrade Ruby to 3.1.2.
Also fix a call to `Time.utc` that fails in Ruby 3.1.2 (can't pass a
string to Time.utc).
2022-04-21 21:43:06 -05:00
nonamethanks
e1b9166a56 Sources: do not use an empty else in case blocks 2022-04-22 03:53:18 +02:00
nonamethanks
3b055138ff Fix normalization for fandom sources 2022-04-22 03:27:05 +02:00
nonamethanks
e6cb255a7a Foundation: fix some video posts not being extracted
Also adjusts SourceTestHelper to not autogenerate contexts, so that
tests can be launched individually.
2022-04-21 17:54:22 +02:00
evazion
29e0139583 Fix #5135: /count Discord slash command broken with aliases and shortcuts. 2022-04-20 22:30:23 -05:00
evazion
d44bb779fe Merge pull request #5134 from nonamethanks/fix-for-button-fix
Fix regression in 7486836b0f that removed the delete button from pending posts
2022-04-20 16:09:31 -05:00
nonamethanks
e4a24d000f Fix regression in 7486836b0f that removed the delete button from pending posts 2022-04-20 19:33:17 +02:00
evazion
edb67b1b81 Merge pull request #5131 from nonamethanks/fix-mod-action-for-post-ban
Posts: don't try to ban/unban a post that is already banned/unbanned
2022-04-19 16:37:13 -05:00
nonamethanks
234ac98640 Posts: don't try to ban/unban a post that is already banned/unbanned 2022-04-19 21:40:47 +02:00
nonamethanks
c5e6044c23 Anifty: regex fixup for c9227645d9 2022-04-19 15:54:56 +02:00
evazion
fdb259f67d apm: record user preference cookies in apm. 2022-04-19 06:59:24 -05:00
evazion
957076d93b apm: don't record unknown url params in the apm.
Don't record unknown url params that don't come from our app. This
includes typos, url params from userscripts, and weird params from
broken bots, crawlers, or other unknown sources. Indexing too many
params can lead to a mapping explosion.

https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping.html
2022-04-19 06:59:24 -05:00
evazion
c187d56cce apm: record only select http headers in the apm.
Don't record most HTTP request and response headers in the APM, except
for the User-Agent, Referer, Save-Data, X-Forwarded-For, Accept-Language,
and Content-Type headers. Recording every HTTP header for every request
takes up a lot of space and most of them aren't very useful.
2022-04-19 06:59:24 -05:00
evazion
d3294364e1 Merge pull request #5129 from nonamethanks/anifty-support
Add anifty.jp support
2022-04-19 06:54:21 -05:00
evazion
9a62bb673f Merge pull request #5130 from nonamethanks/readd-approve-button
Readd approval button to the sidebar
2022-04-19 06:34:34 -05:00
evazion
14c8146a5d stripe: notify site owner when a chargeback is created.
Send a notification Dmail to the site owner when a chargeback is
created, or when an impending chargeback warning is received.
2022-04-18 19:46:44 -05:00
nonamethanks
7486836b0f Readd approval button to the sidebar
The quick mod buttons are also used in the modqueue so it's not
possible to bind the shift-o shortcut to them, hence the need
for this button.
2022-04-18 19:50:35 +02:00
nonamethanks
c9227645d9 Add anifty.jp support 2022-04-18 16:50:26 +02:00
evazion
a45b18ed36 views: inline notes partial into posts show page. 2022-04-18 04:30:01 -05:00
evazion
dff0b0e48b posts: add back option to undelete posts in sidebar.
This was removed in 1a990d5ab, but that left no way to undelete posts
that haven't been appealed since the quickmod bar isn't shown in that
case.
2022-04-18 04:28:54 -05:00
evazion
089adc085b Fix #5119: /counts/posts ignores aliased tags on betabooru 2022-04-18 00:52:08 -05:00
evazion
6c5dd5ffed tests: fix broken tests. 2022-04-18 00:31:31 -05:00
evazion
cd22c8eb1a posts: eliminate unnecessary favgroup query on posts show page.
Eliminate a SQL query loading the post's favgroups if the current user
is anonymous (and therefore can't have favgroups).
2022-04-18 00:18:54 -05:00
evazion
5050ca527d posts: eager load uploader and media asset on posts show page.
Eliminate a couple SQL queries by loading the uploader and media asset
in the same query as the post.
2022-04-18 00:18:54 -05:00
evazion
436f3eaf5d posts: eliminate redundant tag load on posts show page.
Eliminate an unnecessary `SELECT name, post_count, category FROM tags WHERE name IN (?)`
query on the post show page. This query was used by `humanized_essential_tag_string`
when generating the page title. This query was redundant because we already use
another query to load the tag list with `SELECT * FROM tags WHERE name IN (?)`.
2022-04-18 00:18:50 -05:00
evazion
219a0959a5 saved searches: fix to use new PostQuery class. 2022-04-17 23:20:22 -05:00
evazion
ca5dd61728 post queries: optimize zero tag and single tag searches.
Avoid going through the full post query parser for empty searches or
simple single-tag searches.
2022-04-17 23:20:22 -05:00
evazion
652db0cd9f post queries: show full error message on error page. 2022-04-17 23:20:22 -05:00
evazion
eca0ab04f7 post queries: raise error on invalid searches.
Raise an error if the search is invalid for one of the following reasons:

* It contains multiple conflicting order: metatags (e.g. `order:score order:favcount` or `ordfav:a ordfav:b`).
* It contains a metatag that can't be used more than once: (e.g. `limit:5 limit:10`, `random:5 random:10`).
* It contains a metatag that can't be negated (e.g. `-order:score`, `-limit:20`, or `-random:20`).
* It contains a metatag that can't be used in an OR clause (e.g. ` touhou or order:score`, `touhou or limit:20`, `touhou or random:20`).
2022-04-17 23:20:22 -05:00
evazion
c45d1d42c2 post queries: fix parsing of trailing parentheses.
Fix queries like `(fate_(series) saber)` being parsed as `fate_(series` + `saber)`
instead of `fate_(series)` + `saber`.

This is pretty hacky. We assume that parentheses in tags are balanced.
So the rule is that trailing parentheses are part of the tag as long as
they're balanced, and not part of the tag if they're unbalanced.
2022-04-17 23:20:22 -05:00
evazion
5f1c296011 tags: don't allow tags with unbalanced parentheses.
Don't allow tags to have unbalanced parentheses, except for a few
emoticon tags as special exceptions to the rule.
2022-04-17 23:20:22 -05:00
evazion
3e8e33e663 post queries: fix handling of '~' operator.
Fix queries like `(~a ~b) (~c ~d)` being handled like `~a ~b ~c ~d`.
Caused by trimming AND nodes from the tree before rewriting the '~'
operator, which caused `~a` terms to be incorrectly lifted out of
subexpressions.
2022-04-17 23:20:22 -05:00
evazion
c4e1f5bf3d posts: fix post search count logging. 2022-04-17 23:20:22 -05:00
evazion
7584578166 posts: fix post query logging. 2022-04-17 23:20:22 -05:00
evazion
af183467b6 post queries: switch to new post search engine.
Switch to the post search engine using the new PostQuery parser. The new
engine fully supports AND, OR, and NOT operators and grouping expressions
with parentheses.

Highlights:

New OR operator:

* `skirt or dress` (same as `~skirt ~dress`)

Tags can be grouped with parentheses:

* `1girl (skirt or dress)`
* `(blonde_hair blue_eyes) or (red_hair green_eyes)`
* `~(blonde_hair blue_eyes) ~(red_hair green_eyes)` (same as above)
* `(pantyhose or thighhighs) (black_legwear or brown_legwear)`
* `(~pantyhose ~thighhighs) (~black_legwear ~brown_legwear)` (same as above)

Metatags can be OR'd together:

* `user:evazion or fav:evazion`
* `~user:evazion ~fav:evazion`

Wildcard tags can combined with either AND or OR:

* `black_* white_*` (find posts with at least one black_* tag AND one white_* tag)
* `black_* or white_*` (find posts with at least one black_* tag OR one white_* tag)
* `~black_* ~white_*` (same as above)

See 4c7cfc73 for more syntax examples.

Fixes #4949: And+or search?
Fixes #5056: Wildcard searches return unexpected results when combined with OR searches
2022-04-17 23:20:22 -05:00
evazion
703fd05025 favgroups: don't allow favgroups to be named 'any' or 'none'.
'any' and 'none' are now reserved keywords for the favgroup: metatag.

Also add a fix script to rename existing favgroups.
2022-04-17 23:17:18 -05:00
evazion
db49a4fbff views: inline partials in posts/show layout.
Reduce the number of spans reported to the APM.
2022-04-17 23:03:00 -05:00
evazion
604b6ce547 views: inline partials into default layout.
Inline the page footer, news updates, the ban notice, and the user
verification notice into the default layout. This is a micro
optimization to reduce the number of spans reported to the APM.
2022-04-17 23:00:31 -05:00
evazion
d19e307e69 Merge pull request #5125 from nonamethanks/booth-support
Add Booth support
2022-04-17 23:00:14 -05:00
evazion
adbef1b869 Merge pull request #5113 from nonamethanks/prune-disapprovals-on-queue-enter
Posts: prune disapprovals on new appeal or flag
2022-04-17 22:57:49 -05:00
evazion
e35bbb8bc8 Merge pull request #5120 from nottalulah/favgroup-any
favgroups: allow favgroup:any/none searches
2022-04-17 22:55:55 -05:00
evazion
4f684044e3 Merge pull request #5114 from nonamethanks/editable-post-disapprovals
Allow post disapprovals to be edited
2022-04-17 22:54:57 -05:00