Fix a division by zero when calculating the upvote ratio in the votes tooltip.
This could happen if the post had votes according to the post_votes
table, but didn't have any votes according to the up_score and
down_score. This should never happen, yet it did happen for post 2248039
and post 2959269.
Make the `order:random` metatag truly randomize the search. Add a
`random:N` metatag that returns up to N random posts, like what
`order:random` did before.
`order:random` now returns the entire search in random order. Before it
just returned a pageful of pseudorandom posts. This will be more
accurate for small searches, but slower for large searches. If
`order:random` times out, try `random:N` instead.
The `random:N` metatag returns up to N pseudorandom posts. This is
faster than `order:random` for large searches, but for small searches,
it may return less than N posts, and the randomness may be biased. Some
posts may be more likely than others to appear. N must be between 0 and
200.
Also, `/posts?tags=touhou&random=1` now redirects to `/posts?tags=touhou+random:N`.
Before the `random=1` param acted like a free `order:random` tag; now it
redirects to a `random:N` search, so it counts against your tag limit.
Optimize metatag searches involving usernames, including user:,
approver:, appealer:, commenter:, upvoter:, etc.
Do `User.find_by_name` instead of `User.name_matches` because this
fetches the user upfront instead of doing it inside a subquery. Using a
subquery makes the SQL more complicated and leads to worse query plans.
This especially helps searches involving multiple username metatags.
Allow admins to remove votes on posts. This is for fixing vote abuse.
Votes can be removed by going to the vote list on the /post_votes page,
or by clicking on a post's score, then using the "Remove" option in the
"..." dropdown menu next to the vote.
Votes are soft-deleted - they're marked as deleted in the database, but
not fully deleted. Removed votes are only visible to admins, not to
regular users. When a vote is removed by an admin, it leaves a mod
action.
Technically it's possible to undelete votes, but there's no UI for it.
Optimize searches using the `search[user_name]=...` URL parameter. If
we're not doing a wildcard search, then do a regular user lookup, which
generates better SQL.
* Add a gap between thumbnails on mobile.
* Adjust CSS for scores and vote buttons.
* Include "Private favorites" as an incentive on the user upgrade page.
* Fix vote buttons not being visible beneath thumbnails on mobile.
* Fix the "Show scores" link not preserving the current page number.
* Fix vote buttons being unintentionally enabled for all thumbnails by default.
* Fix banned and restricted users being able to favorite posts by
tagging them with `fav:self`.
* Fix search engines being able to crawl /posts?view=score pages.
* Fix broken tests.
Changes:
* Make it so you can click or hover over a post's favorite count to see
the list of public favorites.
* Remove the "Show »" button next to the favorite count.
* Make the favorites list visible to all users. Before favorites were
only visible to Gold users.
* Make the /favorites page show the list of all public favorites,
instead of redirecting to the current user's favorites.
* Add /posts/:id/favorites endpoint.
* Add /users/:id/favorites endpoint.
This is for several reasons:
* To make viewing favorites work the same way as viewing upvotes.
* To make posts load faster for Gold users. Before, we loaded all the
favorites when viewing a post, even when the user didn't look at them.
This made pageloads slower for posts that had hundreds or thousands of
favorites. Now we only load the favlist if the user hovers over the favcount.
* To make the favorite list visible to all users. Before, it wasn't
visible to non-Gold users, because of the performance issue listed above.
* To make it more obvious that favorites are public by default. Before,
since regular users could only see the favcount, they may have
mistakenly believed other users couldn't see their favorites.
Add upvote and downvote buttons beneath thumbnails on the post index page.
This is disabled by default. To enable it, click the "..." menu in the top
right of the page, then click "Show scores".
This is currently a per-search setting, not an account setting. If you
enable it in one tab, it won't be enabled in other tabs.
Make private favgroups a Gold-only option. This is for consistency with
private favorites and upvotes being Gold-only options.
Existing Members with private favgroups are allowed to keep them, as
long as they don't disable privacy. If they disable it, then they can't
re-enable it again without upgrading to Gold first.
Make private favorites and upvotes a Gold-only account option.
Existing Members with private favorites enabled are allowed to keep it
enabled, as long as they don't disable it. If they disable it, then they
can't re-enable it again without upgrading to Gold first.
This is a Gold-only option to prevent uploaders from creating multiple
accounts to upvote their own posts. If private upvotes were allowed for
Members, then it would be too easy to use fake accounts and private
upvotes to upvote your own posts.
* Allow Member-level users to vote.
* Don't allow Banned or Restricted users to create favorites any more.
Banned and Restricted users aren't allowed to upvote or favorite any
more to prevent sockpuppet accounts from upvoting even after they're
banned.
Make upvotes public the same way favorites are public:
* Rename the "Private favorites" account setting to "Private favorites and upvotes".
* Make upvotes public, unless the user has private upvotes enabled. Note
that private upvotes are still visible to admins. Downvotes are still
hidden to everyone except for admins.
* Make https://danbooru.donmai.us/post_votes visible to all users. This
page shows all public upvotes. Private upvotes and downvotes are only
visible on the page to admins and to the voter themselves.
* Make votes searchable with the `upvote:username` and `downvote:username`
metatags. These already existed before, but they were only usable by
admins and by people searching for their own votes.
Upvotes are public to discourage users from upvoting with multiple
accounts. Upvote abuse is obvious to everyone when upvotes are public.
The other reason is to make upvotes consistent with favorites, which are
already public.
Add `upvotes:N`, `downvotes:N`, `order:upvotes`, `order:downvotes`,
`order:upvotes_asc`, `order:downvotes_asc` metatags.
In the API, the field is called up_score / down_score. Here it's called
`upvotes` and `downvotes` because this should be easier to understand
for end users.
Note that internally, `down_score` is negative. A post that matches
`downvotes:>5` will have down_score < -5 internally.
Fix certain artist commentaries for foundation.app containing scrambled
characters. Apparently caused by the Nokogiri HTML5 parser not handling
UTF-8 input correctly when the encoding isn't explicitly set to UTF-8.
* Fix /proc not being mounted read-only.
* Fix other read-only directories not actually being mounted read-only.
This was because the mount(2) system call ignores mount flags when
creating a bind mount. The solution is to bind mount the directory
first, then remount it as read-only second.
Known bug: submounts still don't get mounted as read-only. That is, if
we're mounting /usr as read-only, and /usr has a submount of /usr/local,
then /usr/local won't get mounted as read-only.
Fix thumbnails incorrectly showing a duration of "0:00" when a video had
a duration between 59.5 seconds and 60.0 seconds. This happened because
of incorrect rounding - the seconds value was rounded up, but the
minutes value wasn't.
Examples: https://danbooru.donmai.us/posts?tags=duration:59.5...60.0+status:any
Add a Sandbox class for running untrusted external programs like ffmpeg
or exiftool inside a sandbox. This uses Linux namespaces to run the
process in an isolated container, much like a Docker container. Unlike a
Docker container, we can use it to sandbox programs when Danbooru itself
is already running inside a Docker container.
This is also more restrictive than Docker in several ways:
* It has a system call filter that is more restrictive and more
customizable than Docker's filter by default. Even if the process
breaks out of the container, the syscall filter will limit what it can
do, even if it escalates to root.
* It blocks the use of setuid binaries, so the process can't use things
like sudo to escalate to root inside the sandbox.
* It blocks all network access inside the sandbox by default.
* All files in the container are read-only by default. The sandboxed
process can only communicate by writing to stdout.
See app/logical/sandbox.rb for more details.
This isn't actually enabled yet. It will be rolled out progressively to
ensure it doesn't break things.
Block Googlebot from crawling certain slow useless URLs. Sometimes
Googlebot tries to crawl old source:<url>, approver:<name>, and
ordfav:<name> searches in bulk, which tends to slow down the site because
things like source:<url> are inherently slow, and because Google spends
hours at a time crawling them in parallel. This is despite the fact that
these links are already marked as nofollow and noindex, and source:<url>
links were removed from posts a long time ago to try to stop Google from
crawling them.
Add a Ruby wrapper library around the libseccomp library. Seccomp is
used to restrict the syscalls a program can make. See comments in
app/logical/seccomp.rb for further details.
This is not used for anything yet. It's simply adding part of the
sandboxing infrastructure for later use.
Send all logs to stderr by default instead of stdout. Fixes a problem
where parsing the output of sandboxed commands could fail, because they
could contain Rails log messages in their stdout.
When we run a command in a sandbox, we call fork+exec to run the command
in the background so we can capture its output. If Rails prints
anything to stdout between the fork and exec calls, then it will be
inadvertently captured along with the command's output. This will break
parsing of the command's output. This can happen if warning messages are
printed by Rails while setting up the sandbox between the fork and exec
calls.
Writing to stderr is also more correct, since stdout is buffered by
default, which means logs could potentially be lost if the process dies
unexpectedly before the buffers are flushed. Stderr is unbuffered by
default, which means logs will always be output immediately.
Only load the `listen` and `solargraph` gems in the development
environment, not the test environment. The `listen` gem automatically
spawns background threads to listen for file changes, in order to
automatically reload code when files change, which we don't want or need
in test mode. These threads can interfere with sandboxing, because they
prevent us from being able to call unshare(2) (which can only be called
from a single-threaded process).
Bug: If a media asset got stuck in the 'processing' state during upload,
then it would stay stuck forever and the file couldn't be uploaded again
later.
Fix: Mark stuck assets as failed before raising the "Upload failed"
error. Once the asset is marked as failed, it can be uploaded again
later. Also, only wait for assets to finish processing if they were
uploaded less than 5 minutes ago. If a processing asset is more than 5
minutes old, consider it stuck and mark it as failed immediately.
Assets getting stuck in the processing state is a 'this should never
happen' error. Normally if any kind of exception is raised while
uploading the asset, the asset will be set to the 'failed' state. The
only way an asset can get stuck is if it fails and the exception handler
doesn't run, or the exception handler itself fails. This might happen if
the process is unexpectedly killed, or possibly if the HTTP request
times out and a TimeoutError is raised at an inopportune time. See below
for discussion of issues with Timeout.
[1]: https://vaneyckt.io/posts/the_disaster_that_is_rubys_timeout_method/
[2]: https://jvns.ca/blog/2015/11/27/why-rubys-timeout-is-dangerous-and-thread-dot-raise-is-terrifying/
[3]: https://adamhooper.medium.com/in-ruby-dont-use-timeout-77d9d4e5a001
[4]: https://ruby-doc.org/core-3.0.2/Thread.html#method-c-handle_interrupt-label-Guarding+from+Timeout-3A-3AError
Change the rules for automatically retiring aliases and implications:
* Retire aliases to tags that are empty, or that are for a general or
artist tag that hasn't received any new posts in the last two years.
* Retire implications from tags that are empty.
* Don't retire aliases or implications for character, copyright, or
meta tags any more, unless the tags are empty.
Fix `https://danbooru.donmai.us/artists.json?expires_in=300` failing with
an `'300' is not a valid duration` error. This call pattern is used by the
Translate Pixiv Tags userscript.
Caused by a5ed8c72, which changed the `age:N` metatag to require time
units, but this inadvertently changed the `expires_in` parameter to
require them too.
Using `expires_in` without time units is deprecated and will be removed
in the future.
Make it possible to reapprove failed BURs that removed aliases or
implications.
Before if a BUR failed midway through, and we tried to reapprove it,
then it would fail when it got to a `remove alias` line because the
alias had already been removed. Now we keep going if we try to remove an
alias or implication that has already been removed.
Change the /bulk_update_requests page to show the newest BURs first
instead of pending BURs first. This is more consistent with index pages
on the rest of the site, which normally default to newest first. Fixes
an issue where failed BURs would be shown first forever unless they were
manually approved or rejected.
Fix the video duration not being shown on thumbnails on the
https://danbooru.donmai.us/comments page.
BUG: this introduces duplicate HTML ids on the comments page. Post
thumbnails and post comment containers both have the same html ID.
* Change `age:` metatag to require time units. This means e.g.
`age:<600` no longer works; instead you have to say `age:<600sec`.
* Allow time units in the `age:` metatag to be abbreviated as long as
they're unambiguous. This means `age:<60sec`, `age:<5min`, and
`age:<5mon` now work, in addition to `age:<60s` and `age:<60seconds`.
* Allow the `ratio:` metatag to be written like `ratio:16/9` in addition
to `ratio:16:9`.
* Fix invalid date searches like `date:foo` or `date:05-15-2021`
to return nothing instead of raising an "undefined method
'beginning_of_day' for nil" exception. (`date:05-15-2021` is invalid
because it's parsed as DD-MM-YYYY).
* Fix invalid searches like `score:foo`, `ratio:foo`, and `mpixels:foo`
to return nothing instead of being treated like `score:0`, `ratio:0`,
`mpixels:0`.
* Fix `age:<60m` to return nothing instead of silently being treated
like `age:<60seconds`.
* Fix `age:foo` to return nothing instead of silently being treated like
`age:0d` (return all uploads from today).
Fixes#4389.
The favorites table is too big and dumping it tends to time out. Then
the job keeps retrying even though it always fails, then multiple
instances of the job build up in the job queue because the old jobs
never finish.
Fix the `enable_seo_post_urls` config option not being respected. This
option controls whether filenames in image URLs contain the tags. This
option requires URLs rewrites in Nginx to work so it's disabled by
default.
Refactor StorageManager to remove all image URL generation code. Instead
the image URL generation code lives in MediaAsset.
Now StorageManager is only concerned with how to read and write files to
remote storage backends like S3 or SFTP, not with how image URLs should
be generated. This way the file storage code isn't tightly coupled to
posts, so it can be used to store any kind of file, not just images
belonging to posts.
Fix it so that when a post is expunged, the media asset is also marked
as expunged. This way the files will be deleted, but the media asset
will still remain as a record of what was expunged. The media asset will
have the md5, width, height, file ext, and file size of the deleted file.
Switch the font to 11px bold Arial. This is more compact and more
readable than 9px Tahoma. Also add a slight border radius and margins
around the indicator to make it stand out from the edge of the image.
Fix various elements to use standard font sizes instead of ad-hoc sizes.
Noticeable changes:
* Tags in autocomplete are slightly smaller.
* The favorite heart icon on posts is slightly smaller.
* Pool titles on thumbnails in the pool gallery page are slightly bigger.
* The page footer is slightly smaller.
* Timestamps on comments and forum posts are very slightly smaller.
* "Pending"/"approved"/"rejected" labels on forum posts are very slightly smaller.
Use rem units for font sizes so that font sizes are relative to the root
<html> element, not the parent element.
Fixes an issue where the video duration indicator would be too small on
parent/child thumbnails in post show pages. This was because of nesting
issues with em units. Em units are relative to their parent element, so
if you had a parent element with a font size of 0.8em, and a child
element with a font size of 0.8em, then the final computed font size
would be 0.8*0.8 = 0.64em.
Show the length of videos and animated posts in the thumbnail. The
length is shown the top left corner in MM:SS format. This replaces the
play button icon.
Show a speaker icon instead of a music note icon for posts with sound.
Doing this requires doing `.includes(:media_asset)` in a bunch of
places to avoid N+1 queries when we access the post's duration.
Mark assets that have missing files as expunged. This happened with
uploads that were abandoned and had their files deleted, but that didn't
destroy their media asset record.
Fixes an issue where uploads could have missing files because someone
resumed an abandoned upload that had its files deleted.
Lower the priority of the populate saved search job. This is so that
large numbers of saved searches don't overwhelm the job queue and
prevent higher priority jobs from running.
Don't delete replaced files after 30 days. There are only about 30k
replacements in total, so the cost of keeping replaced files is
negligible. It was also wrong because the media asset wasn't destroyed
too, so there were active media assets with missing files.
Don't destroy Pixiv Ugoira frame data when the media asset is destroyed.
This is wrong because when uploads were pruned, it could delete the
frame data of an active post.
Add a md5 uniqueness constraint on media assets to prevent duplicate
assets from being created. This way we can guarantee that there is one
active media asset per uploaded file.
Also make it so that if two people are uploading the same file at the
same time, the file is processed only once.
* Make it so replacing a post doesn't generate a dummy upload as a side effect.
* Make it so you can't replace a post with itself (the post should be regenerated instead).
* Refactor uploads and replacements to save the ugoira frame data when
the MediaAsset is created, not when the post is created. This way it's
possible to view the ugoira before the post is created.
* Make `download_file!` in the Pixiv source strategy return a MediaFile
with the ugoira frame data already attached to it, instead of returning it
in the `data` field then passing it around separately in the `context`
field of the upload.
Move more of the file-handling logic from UploadService and
StorageManager into MediaAsset. This is part of refactoring posts and
uploads to allow multiple images per post.
Fix how the duration of videos and animated GIFs / PNGs is calculated.
If we can't determine the duration from the file metadata, then play the
entire video or animation back using FFmpeg and scrape the duration and
frame count.
This is necessary for things like WebM files where the duration metadata
is optional, or animated GIFs and PNGs that don't have a duration field
in the metadata, only a frame count and a sequence of frame delays.
Refactor full-text search on several tables (comments, dmails,
forum_posts, forum_topics, notes, and wiki_pages) to use to_tsvector
expression indexes instead of dedicated tsvector columns. This way
full-text search works the same way across all tables.
API changes:
* Changed /wiki_pages.json?search[body_matches] to match against only
the body. Before `body_matches` matched against both the title and the body.
* Added /wiki_pages.json?search[title_or_body_matches] to match against
both the title and the body.
* Fixed /dmails.json?search[message_matches] to match against both the
title and body when doing a wildcard search. Before a wildcard search
only matched against the body.
* Added /dmails.json?search[body_matches] to match against only the dmail body.
Make it so that when a database call inside a `with_timeout` block times
out, the error logged to New Relic is marked as expected. This is so
that expected timeouts, such as timeouts when calculating search counts
or timeouts when generating related tags for the sidebar, don't count
against the error rate.
Drop the final dependency on the Postgres test_parser extension.
We also have to remove references to test_parser in the migration where
it was first defined, otherwise replaying all migrations from the
beginning will fail. Replaying all migrations from the beginning
normally isn't done except in testing.
After this, it should be possible to use a vanilla install of Postgres
with Danbooru. It's still recommended to use Danbooru's Docker image for
Postgres (https://ghcr.io/danbooru/postgres), as other Postgres extensions
may be necessary in the future.
Restructure the Dockerfile and the CSS/JS files so that we only rebuild
the CSS and JS when they change, not on every commit.
Before it took several minutes to rebuild the Docker image after every
commit, even when the JS/CSS files didn't change. This also made pulling
images slower.
This requires refactoring the CSS and JS to not use embedded Ruby (ERB)
templates, since this made the CSS and JS dependent on the Ruby
codebase, which is why we had to rebuild the assets after every Ruby
change.
evazion
evazion