Commit Graph

10854 Commits

Author SHA1 Message Date
evazion
e57e38b35f tests: fix broken tests. 2021-12-08 03:01:54 -06:00
evazion
08bcd51ac8 media assets: add rake task to validate thumbnails.
Update `bin/rails danbooru:images:validate` to work on media assets
instead of posts, and to check that all thumbnails exist.
2021-12-07 18:01:21 -06:00
evazion
8669edd93f artists: add mega.nz to artist finder blacklist. 2021-12-07 04:12:52 -06:00
evazion
dab31a3ef0 media assets: fix exception when generating thumbnails for videos/ugoiras. 2021-12-06 19:01:34 -06:00
evazion
3d4d8ae2ae media assets: fix thumbnail backfill script to ignore Flash files.
We can't generate thumbnails for Flash files, so ignore them.
2021-12-05 21:48:57 -06:00
evazion
c22f7b799b media assets: fix error when generating thumbnails for corrupt files.
Fix an error being raised when trying to generate thumbnails for corrupt
files. If the original image is corrupt, then ignore any errors and let
libvips try to generate a thumbnail as best it can. This will usually
result in an incomplete thumbnail.
2021-12-05 21:46:14 -06:00
evazion
853569701b media assets: fix error when thumbnailing 16-bit images.
Fix an error caused when trying to generate thumbnails for 16-bit
PNG images.
2021-12-05 21:36:20 -06:00
evazion
d47154a9a6 media assets: fix typo in thumbnail backfill script. 2021-12-05 19:04:21 -06:00
evazion
ad49a10147 media assets: fix bug in thumbnail generation.
Fix thumbnail generation throwing a NoMatchingPatternError.
2021-12-05 19:04:17 -06:00
evazion
bcc773390b media assets: add script to backfill new thumbnail sizes. 2021-12-05 16:41:32 -06:00
evazion
396062869a posts: fix thumbnails on comment and modqueue pages.
Fix thumbnails being fit-to-width instead of a fixed size on the
/comments and /modqueue pages, which caused the columns to be misaligned.
2021-12-05 16:39:24 -06:00
evazion
9cb70fa632 posts: add 720x720 thumbnail size.
This is used to provide higher resolution thumbnails for high pixel
density displays, such as phones or laptops. If your screen has a 2x
pixel density ratio, then 360x360 thumbnails will be rendered at 720x720
resolution.

We use WebP here because it's about 15% smaller than the equivalent
JPEG, and because if a device has a high enough pixel density to use
this, then it probably supports WebP.

720x720 thumbnails average about 36kb in size, compared to 20.35kb for
360x360 thumbnails and 7.55kb for 180x180 thumbnails.
2021-12-05 09:19:29 -06:00
evazion
92906f7d39 posts: fix recommended posts layout.
Fix recommended posts page to use new post gallery component.
2021-12-05 03:37:56 -06:00
evazion
40568a9843 posts: fix parent/child post layout.
Fix regression causing parent/child posts to be laid out as a grid
instead of as a single row of posts.
2021-12-05 02:52:08 -06:00
evazion
2e6f480d07 pools: fix pool gallery page layout.
Fix the /pools/gallery page layout being broken by 8841de68a.

This required refactoring the PostGalleryComponent to take a set of
PostPreviewComponents instead of a set of Posts.

The upshot is that it's technically possible to have adjustable
thumbnail sizes on the pool gallery page now (although this is not yet
exposed in the UI).
2021-12-05 02:52:07 -06:00
evazion
8841de68ac posts: add adjustable thumbnail sizes (#4932).
Add a menu in the top right of the post index page that lets you select
the thumbnail size.

This menu is currently hidden until the new thumbnails have been generated.

On desktop, there are five thumbnail sizes:

* Small:    150x150 (https://danbooru.donmai.us/posts?size=150)
* Medium:   180x180 (https://danbooru.donmai.us/posts?size=180)
* Large:    225x225 (https://danbooru.donmai.us/posts?size=225)
* Huge:     270x270 (https://danbooru.donmai.us/posts?size=270)
* Gigantic: 360x360 (https://danbooru.donmai.us/posts?size=360)

On mobile, there are four sizes:

* Small:  150x150 / 3 posts per row (https://danbooru.donmai.us/posts?size=150)
* Medium: 180x180 / 2 posts per row (https://danbooru.donmai.us/posts?size=180)
* Large:  225x225 / 2 posts per row (https://danbooru.donmai.us/posts?size=225)
* Huge:   360x360 / 1 posts per row (https://danbooru.donmai.us/posts?size=360)

There are two extra sizes that aren't listed in the menu:

* 225x360 (https://danbooru.donmai.us/posts?size=225w)
* 270x360 (https://danbooru.donmai.us/posts?size=270w)

These sizes are good for tall thumbnails, but not so much for wide
thumbnails. They aren't listed because in practice they're a bit too big.

The 225x225 and 270x270 sizes are really just 360x360 thumbnails scaled
down in HTML. This means 225x225 and 360x360 thumbnails both use the
same amount of bandwidth.

Thumbnail size is currently a per-search option, not a persistent
account-level setting.

This changes the HTML structure of thumbnails somewhat, so this may
break userscripts and custom CSS.
2021-12-04 06:30:29 -06:00
evazion
17537084fe posts: generate 180x180px and 360x360px thumbnails (#4932).
Add two new thumbnail sizes. These new thumbnail sizes are generated on
upload, but not used yet.
2021-12-02 23:42:44 -06:00
evazion
9e77ff4948 posts: disable cropped (square) thumbnails on mobile.
Disable cropped (square) thumbnails on mobile. Use regular uncropped
thumbnails instead.

This is for a few reasons:

* It made it harder to support multiple thumbnail sizes or file formats,
  since we need a cropped and uncropped version for every size and format.
* The cropping algorithm wasn't that great and sometimes cropped out
  important parts of the image.
* The thumbnail type (cropped or uncropped) was automatically chosen
  clientside based on the user's screen size, which made certain things
  like adjustable thumbnail sizes more difficult because we didn't know
  which thumbnail type the user actually had.

This may return again as an option in the future, but for now it's disabled.
2021-12-02 23:32:23 -06:00
evazion
e8f9e3ab28 posts: show 2 posts per row on mobile instead of 3.
Three thumbnails per row was pretty tight for most phones since the most
common phone resolution is 360x640. Two thumbnails per row lets us have
thumbnails up to 180x180 in size.
2021-12-02 23:32:17 -06:00
evazion
74e28e14b8 posts: factor out post gallery component.
Factor out thumbnail galleries into a PostGallery component.

This changes the html structure so that post galleries on all pages are
always wrapped in a `.posts-container` class. This fixes an issue with
thumbnails on the pool show page not being aligned correctly on mobile,
like they are on the post index page. This also affected thumbnail
galleries on other pages, like wiki pages and user profiles.
2021-12-02 22:18:00 -06:00
evazion
15657aad6d emails: update email domain whitelist. 2021-12-02 20:49:20 -06:00
evazion
6fc0854b4c Remove StorageManager::SFTP.
Remove the SFTP file storage backend. Downstream users can use either
sshfs (which is what Danbooru now uses in production) or rclone instead.
The Ruby SFTP gem was much slower than sshfs.
2021-12-01 23:46:20 -06:00
evazion
5bf4e31c25 users: fix incorrect vote counts on user profile pages.
Deleted comment votes and post votes were being incorrectly counted.
2021-12-01 04:47:47 -06:00
evazion
f86e16cfbd MediaFile: allow generating thumbnails for corrupt files.
We need this so we can regenerate thumbnails for old posts with
corrupted images.
2021-12-01 04:45:26 -06:00
evazion
2ffe973275 MediaFile: add support for creating .webp and .avif thumbnails.
These other formats aren't actually generated during upload, but support
for creating them is there.

Also tune the parameters for generating JPEGs:

* Use Q=85 instead of Q=90 because Q=85 enables 4:2:0 chroma
  subsampling, while Q=90 doesn't use subsampling. Subsampling reduces
  filesize by ~30% in most cases. It does reduce quality for certain
  images, particularly for images with lots of bright red, but in most
  cases the quality difference isn't noticeable.

* Enable several MozJPEG-specific options, including trellis
  quantization and scan optimization. These reduce filesize without
  reducing quality, at the cost of slower encoding times.
2021-12-01 04:45:26 -06:00
evazion
e5ba6d4afc MediaFile: fix thumbnail dimension calculation.
Calculate the dimensions of thumbnails ourselves instead of letting
libvips calculate them for us. This way we know the exact size of
thumbnails, so we can set the right width and height for <img> tags. If
we let libvips calculate thumbnail sizes for us, then we can't predict
the exact size of thumbnails, because sometimes libvips rounds numbers
differently than us.
2021-12-01 04:45:26 -06:00
evazion
c2e6202da6 Fix #4920: Wrong color for certain samples.
The problem was that we were stripping color profiles from thumbnails,
but we weren't setting `export_profile: "srgb"` to convert images to
sRGB first. This resulted in wrong colors for images with non-sRGB color
profiles, such as Adobe RGB.

The fix is to convert images to sRGB when possible, while leaving CMYK
and greyscale images alone. We leave CMYK images alone because we can't
convert CMYK to sRGB without losing color. We leave greyscale images
alone if they don't have a color profile, that way they stay as
one-channel greyscale (or two-channel greyscale, in case of alpha)
instead of being converted to three-channel sRGB. However, if a
greyscale image has a color profile, then we have to convert to sRGB,
otherwise the colors would be wrong when we strip the profile.

We also have to set the import profile, otherwise images with broken
embedded color profiles won't have a fallback profile and may get
incorrect colors. In this case we also have to be careful, because we
can't specify an sRGB fallback for greyscale or CMYK images.
2021-12-01 04:45:26 -06:00
evazion
c1a37d9577 docker: set MALLOC_ARENA_MAX=2.
Setting MALLOC_ARENA_MAX=2 is claimed to reduce memory bloat caused by
fragmentation. Jemalloc 3.x is also claimed to reduce memory bloat, but
modern distros only ship Jemalloc 5.x, and supposedly only 3.x works.

https://devcenter.heroku.com/articles/tuning-glibc-memory-behavior
https://bugs.ruby-lang.org/issues/14718
https://www.speedshop.co/2017/12/04/malloc-doubles-ruby-memory.html
https://www.joyfulbikeshedding.com/blog/2019-03-14-what-causes-ruby-memory-bloat.html
2021-11-28 22:45:50 -06:00
evazion
9ec93e4dc7 docker: replace libjpeg-turbo with mozjpeg.
Build vips with mozjpeg instead of libjpeg-turbo. Mozjpeg is slower but
has better compression.

https://github.com/mozilla/mozjpeg
https://libjpeg-turbo.org/About/Mozjpeg
https://siipo.la/blog/is-webp-really-better-than-jpeg
https://www.ctrl.blog/entry/webp-avif-comparison.html
http://web.archive.org/web/20151204210024/http://people.mozilla.org/~josh/lossy_compressed_image_study_july_2014/
2021-11-28 17:58:43 -06:00
evazion
2c63fcf081 docker: upgrade Ruby to 3.0.3.
Upgrade bootsnap to 1.9.3 too because Ruby 3.0.3 has a bug that causes
Rails to fail to boot when bootsnap is enabled. Bootsnap 1.9.3 works
around this bug.

Also add libgmp to build with bignum support.
2021-11-28 17:58:43 -06:00
evazion
1e6f57a9ef docker: upgrade libvips to 8.12.1; add .avif support. 2021-11-28 05:20:29 -06:00
evazion
59f8320d91 votes: fix bug with tag scripting in show score mode.
Fix a bug where scores would disappear when applying tag scripts in show
scores mode.
2021-11-28 01:26:09 -06:00
evazion
9591fa1302 votes: fix bug with removing votes multiple times.
Fix an issue where if you tried to remove the same vote multiple times,
nothing would happen.

Specifically, if you upvoted a post in one tab, then opened it in a
second tab, then removed your vote in the first tab, then upvoted it
again in the first tab, then tried to remove your vote again in the
second tab, nothing would happen when removing it in the second tab.

This was because we were removing votes by vote ID, which meant that if
a vote had already been removed, removing it again would do nothing
since it was already deleted.
2021-11-25 23:49:18 -06:00
evazion
8e32f656fa votes: fix division by zero in upvote ratio.
Fix a division by zero when calculating the upvote ratio in the votes tooltip.

This could happen if the post had votes according to the post_votes
table, but didn't have any votes according to the up_score and
down_score. This should never happen, yet it did happen for post 2248039
and post 2959269.
2021-11-25 23:35:09 -06:00
evazion
0baca68a37 search: make order:random truly random; add random:N metatag.
Make the `order:random` metatag truly randomize the search. Add a
`random:N` metatag that returns up to N random posts, like what
`order:random` did before.

`order:random` now returns the entire search in random order. Before it
just returned a pageful of pseudorandom posts. This will be more
accurate for small searches, but slower for large searches. If
`order:random` times out, try `random:N` instead.

The `random:N` metatag returns up to N pseudorandom posts. This is
faster than `order:random` for large searches, but for small searches,
it may return less than N posts, and the randomness may be biased. Some
posts may be more likely than others to appear. N must be between 0 and
200.

Also, `/posts?tags=touhou&random=1` now redirects to `/posts?tags=touhou+random:N`.
Before the `random=1` param acted like a free `order:random` tag; now it
redirects to a `random:N` search, so it counts against your tag limit.
2021-11-25 18:14:34 -06:00
evazion
5dc67613e6 search: optimize username metatags.
Optimize metatag searches involving usernames, including user:,
approver:, appealer:, commenter:, upvoter:, etc.

Do `User.find_by_name` instead of `User.name_matches` because this
fetches the user upfront instead of doing it inside a subquery. Using a
subquery makes the SQL more complicated and leads to worse query plans.
This especially helps searches involving multiple username metatags.
2021-11-25 00:40:53 -06:00
evazion
5ba6014a4e votes: fix username tooltips not appearing inside other tooltips.
Fix a bug where, when hovering over a username inside a tooltip, the
username tooltip wouldn't appear.

May break 0bf5ac73?
2021-11-25 00:02:25 -06:00
evazion
a45e6b5cfe Fix #4931: Add popup voter list for comments.
Show the comment's upvote and downvote count when you hover over a
comment's score. For mods, show the list of voters as well.
2021-11-24 22:18:48 -06:00
evazion
594b46a85d tests: fix broken tests. 2021-11-23 23:18:54 -06:00
evazion
810f0b5d6b favorites: fix empty favorites tooltip.
Fix the favorites tooltip being empty when a post has no favorites.
2021-11-23 23:18:54 -06:00
evazion
353e708538 votes: allow admins to remove post votes.
Allow admins to remove votes on posts. This is for fixing vote abuse.

Votes can be removed by going to the vote list on the /post_votes page,
or by clicking on a post's score, then using the "Remove" option in the
"..." dropdown menu next to the vote.

Votes are soft-deleted - they're marked as deleted in the database, but
not fully deleted. Removed votes are only visible to admins, not to
regular users. When a vote is removed by an admin, it leaves a mod
action.

Technically it's possible to undelete votes, but there's no UI for it.
2021-11-23 23:18:54 -06:00
evazion
692f2848f2 Merge pull request #4929 from nonamethanks/fix-autoban-duration
SpamDetector: raise autoban duration to 100 years.
2021-11-22 20:38:55 -06:00
nonamethanks
524eacf4b9 SpamDetector: raise autoban duration to 100 years.
This is to match the max duration usable by mods in the site.
The previous value of 999_999 seconds only amounted to 11 days.
2021-11-22 18:29:46 +01:00
evazion
f151e93615 votes: fix upvote ratio calculation.
Old posts don't have vote records, so we can't use vote records to
calculate the ratio.
2021-11-21 18:06:41 -06:00
evazion
00befc96f0 /comments: add vote buttons beneath thumbnails on /comments page.
Also fix an N+1 queries problem.
2021-11-21 18:04:52 -06:00
evazion
be5173c8dd votes: add is_deleted flag to post_votes table.
Add an is_deleted flag to post_votes so they can be soft-deleted in the future.
2021-11-21 02:36:30 -06:00
evazion
35c97d0836 mode menu: remove upvote/downvote mode.
Remove the upvote and downvote modes from the post mode menu. These are
unnecessary now since you can use the vote buttons beneath the thumbnails.
2021-11-21 02:36:22 -06:00
evazion
6b9e1181e5 search: optimize ?search[user_name]=... searches.
Optimize searches using the `search[user_name]=...` URL parameter. If
we're not doing a wildcard search, then do a regular user lookup, which
generates better SQL.
2021-11-20 03:19:04 -06:00
evazion
c98af3c85a Merge pull request #4923 from danbooru/feat-member-voting
Allow Members to vote
2021-11-20 02:44:16 -06:00
evazion
eda23c719a votes: fixup various minor issues.
* Add a gap between thumbnails on mobile.
* Adjust CSS for scores and vote buttons.
* Include "Private favorites" as an incentive on the user upgrade page.
* Fix vote buttons not being visible beneath thumbnails on mobile.
* Fix the "Show scores" link not preserving the current page number.
* Fix vote buttons being unintentionally enabled for all thumbnails by default.
* Fix banned and restricted users being able to favorite posts by
  tagging them with `fav:self`.
* Fix search engines being able to crawl /posts?view=score pages.
* Fix broken tests.
2021-11-20 02:40:18 -06:00