Commit Graph

11238 Commits

Author SHA1 Message Date
evazion
d770f0bb95 tests: fix broken tests. 2022-03-09 01:14:09 -06:00
evazion
bb4b8619f5 pixiv: fix Source::URL::Pixiv not being included in Source::URL list. 2022-03-09 01:14:09 -06:00
evazion
8a50148823 pixiv: fixup bug with fetching image_urls for bad_id posts.
Fix `image_urls` returning `[nil]` when fetching data for a image URL
that was bad_id. In that case `original_urls` is empty, so we fall back
to using the deleted image URL as-is.
2022-03-09 01:14:09 -06:00
evazion
c989726313 rails: enable remove_deprecated_time_with_zone_name.
Fix this deprecation warning:

    DEPRECATION WARNING: ActiveSupport::TimeWithZone.name has been deprecated
    and from Rails 7.1 will use the default Ruby implementation. You can set
    `config.active_support.remove_deprecated_time_with_zone_name = true` to
    enable the new behavior now.

Triggered by the XML serializer in the API.
2022-03-09 01:14:09 -06:00
evazion
ec190a2d10 Update Rails to 7.0.2.3.
Fixes CVE-2022-21831 (this does not affect Danbooru).

* https://discuss.rubyonrails.org/t/cve-2022-21831-possible-code-injection-vulnerability-in-rails-active-storage/80199
* https://rubyonrails.org/2022/3/8/Rails-7-0-2-3-6-1-4-7-6-0-4-7-and-5-2-6-3-have-been-released
2022-03-09 01:14:09 -06:00
evazion
77c88fd867 Merge pull request #5038 from nonamethanks/remove-redundant-comments
sources: remove redundant comments
2022-03-08 23:28:29 -06:00
evazion
6afb2f8e3c Merge pull request #5037 from nonamethanks/tumblr-refactor
sources: factor out Source::URL::Tumblr
2022-03-08 23:26:30 -06:00
evazion
7d9790bf46 Merge pull request #5036 from NamelessContributor/issue-5006
icon_helper: inline globe_icon. fixes #5006
2022-03-08 23:22:22 -06:00
evazion
cf4b9a6114 Merge pull request #5039 from nonamethanks/simplify-lofter-tag-parsing
Lofter: simplify tag extraction logic
2022-03-08 23:21:57 -06:00
evazion
987f2985d3 Merge pull request #5040 from nonamethanks/fix-weibo-404
Weibo: fix exception for deleted url
2022-03-08 23:08:37 -06:00
evazion
52a2d3418c pixiv: fixup bugs in 1c620f805.
* Fix error when uploading non-ugoira files.
* Fix sample image URLs not being rewritten to full images correctly. We
  have to get the full image URL from the API because given an
  /img-master/ URL, we don't know what the original file extension is.
2022-03-08 23:07:24 -06:00
nonamethanks
c9be77d1f8 Weibo: fix exception for deleted url 2022-03-09 05:31:38 +01:00
evazion
1c620f8055 sources: factor out Source::URL::Pixiv.
* Drop support for preview_urls. This means that IQDB lookups may be
  slower, especially for ugoiras, since we have to download the full
  ugoira now. However, ugoira lookups should produce better results,
  since the ugoira thumbnail chosen by Pixiv wasn't necessarily the same
  as the thumbnail chosen by Danbooru.

* Drop support for uploading single manga pages:

    http://www.pixiv.net/member_illust.php?mode=manga_big&illust_id=18557054&page=2

  Previously uploading an URL like this would only upload a single image
  out of a multi-image work. Now it will upload all images in the work.
  Pixiv no longer supports URLs like this, so we don't either.

* Add support for parsing URLs like this:

    https://i.pximg.net/c/360x360_70/custom-thumb/img/2022/03/08/00/00/56/96755248_p0_custom1200.jpg

  Apparently artists can choose a custom thumbnail now (not like anyone
  will try to upload one though).
2022-03-08 22:17:38 -06:00
evazion
df0bb70486 sources: factor out Source::URL::PixivSketch.
Add upload support for Pixiv Sketch. Fetch tags, commentary, and artist,
and rewrite sample images to full images.

Authentication isn't required. R18 images are hidden in the browser but
visible in the API.
2022-03-08 18:24:12 -06:00
nonamethanks
ff6bfff311 Lofter: simplify tag extraction logic
Now that we have a separate parsing class we can just use it to properly
parse tag urls as well.
2022-03-08 17:01:50 +01:00
nonamethanks
ebd3670076 sources: remove redundant comments
These comments are already present under the parse blocks, so the huge
walls of text before the code are not needed anymore.
2022-03-08 16:56:00 +01:00
nonamethanks
b9c7e467e5 sources: factor out Source::URL::Tumblr
Also adds support for fetching source data from direct image urls when
possible.
2022-03-08 15:06:06 +01:00
NamelessContributor
28343a8e2b icon_helper: inline globe_icon. fixes #5006 2022-03-08 14:19:46 +01:00
evazion
37441d6b1a Merge pull request #5034 from NamelessContributor/fix-spinner-icon
Always add animate-spin class to spinner_icon
2022-03-08 03:30:01 -06:00
evazion
5ce724f1e2 users: remove confirmation when changing username. 2022-03-08 03:29:15 -06:00
evazion
99479dfcf2 users: waive one name change per week rule if user has invalid name. 2022-03-08 03:29:15 -06:00
evazion
f0a5936091 Fix #5035: Double flags being created in certain conditions.
Lock the post to prevent a race condition when validating that the user
hasn't already flagged the post.
2022-03-08 03:29:15 -06:00
NamelessContributor
c9e3c293a9 Always add animate-spin class to spinner_icon
This also fixes the source data and fav button spinners not spinning
after commit 77515915a4
2022-03-08 08:54:45 +01:00
evazion
0713dae49d Merge pull request #5033 from NamelessContributor/fix-hard-tabs
Replace hard tabs with spaces in .rb files
2022-03-08 00:21:23 -06:00
NamelessContributor
5cdbc1d454 Replace hard tabs with spaces in .rb files 2022-03-08 07:11:54 +01:00
evazion
de61e56161 Merge pull request #5032 from nonamethanks/factor-out-weibo
sources: factor out Source::URL::Weibo
2022-03-07 18:31:15 -06:00
evazion
8d28453f17 Merge pull request #5031 from nonamethanks/fix-foundation
Foundation: fix normalization error
2022-03-07 18:28:06 -06:00
evazion
ed98156b72 Merge pull request #5030 from nonamethanks/fix-new-installation-uploads
docker-compose: do db:seed before server start
2022-03-07 18:27:47 -06:00
nonamethanks
d8e2f2ee33 sources: factor out Source::URL::Weibo
Additionally, fixed some broken tests and changed normalization for urls
of album type to point to the mobile version instead, because they're
only visible to logged-in users.
2022-03-07 16:52:43 +01:00
evazion
125cc1d048 users: fix lockout of invalid usernames (525acd17a) 2022-03-07 04:45:37 -06:00
evazion
74d6b4e81e users: don't allow names ending with file extensions.
This is so in the future we can have URLs like https://danbooru.donmai.us/users/evazion
without problems caused by names like https://danbooru.donmai.us/users/evazion.json
2022-03-07 04:39:00 -06:00
evazion
525acd17a5 users: lock out users with invalid names until they change their name.
Forcibly redirect users to the name change page if their name is
invalid. This means user with invalid names can't do anything or view
any pages until they change their name. API requests are still allowed.
2022-03-07 04:36:12 -06:00
evazion
1028bb1c71 Update Ruby gems and Yarn packages. 2022-03-07 04:03:01 -06:00
nonamethanks
d195d30587 Foundation: fix normalization error
Urls like https://foundation.app/@yohan1754/fso/3 would get normalized
like https://foundation.app/@foundation/foundation/3, which was wrong
because it would point to a completely different collection
2022-03-07 06:52:23 +01:00
evazion
4847c6c9dd docker: add foreman to base image.
Add `foreman` to the base Docker image. This way you can do this:

   docker run --rm -it -v $PWD:/danbooru ghcr.io/danbooru/danbooru foreman start

to start everything needed to run Danbooru in development mode (except
for the Postgres database). This will start everything listed in the
Procfile:

   bin/rails server
   bin/good_job start
   bin/rails danbooru:cron
   bin/webpack-dev-server
2022-03-06 23:28:53 -06:00
evazion
0f0f7c768b jobs: use GoodJob in development.
Use the GoodJob job adapter instead of the default Rails async job
adapter in development mode.

The default async adapter runs jobs in a background thread in the
`bin/rails server` process, but this sometimes has problems with jobs
blocking the main server thread. The job queue interface at `/jobs` also
didn't work with this.

This means that now you have to run `bin/good_job start` in development
mode in order to work background jobs. This is required for uploads to
work.
2022-03-06 20:29:56 -06:00
nonamethanks
a34f5f95be docker-compose: do db:seed before server start
This is needed to create DanbooruBot and enable uploads, among other
things.
2022-03-07 03:10:51 +01:00
evazion
a160a3acce users: add stricter username rules.
Add stricter username rules:

* Only allow usernames to contain basic letters, numbers, CJK characters, underscores, dashes and periods.
* Don't allow names to start or end with punctuation.
* Don't allow names to have multiple underscores in a row.
* Don't allow active users to have names that look like deleted users (e.g. "user_1234").
* Don't allow emoji or any other Unicode characters except for Chinese, Japanese, and Korean
  characters. CJK characters are currently grandfathered in but will be disallowed in the future.

Users with an invalid name will be shown a permanent sitewide banner until they change their name.
2022-03-05 01:08:53 -06:00
evazion
ca98e218a1 users: don't bold usernames in comments and forum posts.
Make usernames in comments and forum posts take up less space.
2022-03-01 21:23:21 -06:00
evazion
b4620f561c users: lower max username length to 25 characters.
The median username length is 8 characters. The 99% percentile is 18
characters. The 99.9% percentile is 24 characters. About 750 users have
a name more than 24 characters long.

This doesn't do anything about existing users with long usernames.

Note that this is the length in Unicode codepoints, not grapheme
clusters. Some Unicode characters and emoji may be a single glyph but
composed of multiple codepoints.
2022-03-01 21:23:21 -06:00
evazion
2d4106154b Merge pull request #5025 from NamelessContributor/patch-1
Limit the width of usernames in tables
2022-03-01 20:45:07 -06:00
evazion
68c92b4536 js: fix Javascript failures in Seamonkey/Palemoon.
Fix site Javascript failing to load in Seamonkey, Palemoon, and other
older browsers.

The @alpinejs/morph library uses public instance fields, which is ES2022
syntax not supported in older browsers. This is the code:

    var DomManager = class {
        el = void 0; // `el` is a public instance field
    }
    // => SyntaxError: bad method definition

The fix here is to separate the Alpine code into a separate bundle so
that a failure to load it doesn't cause the rest of the site's
Javascript to fail to load.

A better fix would be to either transpile the @alpinejs/morph library to
ES5 (which seems difficult to do in webpacker), or to fix the library
upstream to not use this syntax.

* https://inspiredwebdev.com/everything-new-in-es2022/
* https://blog.saeloun.com/2021/10/21/ecmacscript-public-instance-fields-and-private-instance-fields.html
* https://caniuse.com/?search=public%20class%20fields
* https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Classes/Public_class_fields#public_instance_fields
2022-03-01 19:40:15 -06:00
evazion
ad3f3fdce3 Fix unqualified column references.
Fix various places to avoid unqualified column references to prevent any
potential ambiguous column errors.
2022-03-01 17:48:16 -06:00
evazion
036341d8ba aliases/implications: fix ambiguous column reference on index page.
Fix searches like this:

    https://danbooru.donmai.us/tag_aliases?search[consequent_tag][category]=1&search[order]=created_at

failing with an ambiguous column reference.
2022-03-01 17:38:46 -06:00
evazion
ece0b1b17d Fix exception on error page when handling PG::AmbiguousColumn error.
Fix an exception on the error page when a controller index action raised
an PG::AmbiguousColumn error because the model `search` method generated
SQL with an ambiguous column reference. In this case the error page
tried to generate data attributes for the <body> tag, but this failed
because evaluating the `current_item` raised an exception again.
2022-03-01 17:09:31 -06:00
NamelessContributor
a43291018a Limit the width of usernames in tables
Fixes the layout issues caused very long usernames in the forum and elsewhere.
2022-03-01 14:59:58 +01:00
evazion
99221af855 ugoiras: fix regression in 7031fd13d.
Fix `Cannot write log file 'ffmpeg2pass-0.log' for pass-1 encoding: Permission denied` error
when uploading ugoira files. Caused by the fact that 2-pass encoding tries to write a log file in
the current directory by default, which fails in production because the default working directory in
the Docker image is /danbooru, which is read-only.
2022-03-01 00:16:55 -06:00
evazion
03560bafc6 uploads: add limit to prevent users from submitting too many uploads at once.
Add a limit so that users can't upload more if they already have more
than 250 images queued for upload.

For example, if you upload a Pixiv post that has 200 images, then you'll
have 200 queued images for upload. This will go down as the images are
processed. If you exceed the limit, then trying to create new uploads
will return an error.

This is to prevent single users from overwhelming the site by uploading
too many images at once, thereby preventing other users from uploading
because the job queue is backed up and can't process new uploads by
other users until existing uploads are finished.
2022-02-28 23:10:12 -06:00
evazion
7031fd13d7 ugoiras: encode .webm samples using VP9 instead of VP8.
Switch the codec for .webm samples from VP8 to VP9. All modern browsers
support VP9 (Safari was the last to add support in ~2020), so it should
be safe to provide only VP9 .webms without a fallback.

VP9 lets us use two-pass encoding, which should offer better compression.

Fixes ugoira samples still having poor quality even after 4c652cf3e.
4c652cf3e tried to remove the max bitrate limit by setting `-b:v 0`, but
this only worked in FFmpeg 4.2. In production Danbooru uses FFmpeg 4.4,
and apparently in 4.4 `-b:v 0` means "use the default max bitrate of
256kb/s" instead of "no bitrate limit".

https://trac.ffmpeg.org/wiki/Encode/VP9
https://developers.google.com/media/vp9/bitrate-modes
https://developers.google.com/media/vp9/settings/vod
http://wiki.webmproject.org/ffmpeg/vp9-encoding-guide
https://www.reddit.com/r/AV1/comments/k7colv/encoder_tuning_part_1_tuning_libvpxvp9_be_more/
2022-02-28 22:02:56 -06:00
evazion
f6a96c632d Fix #5024: URL inflection change broke migrations
Broken in 60a26af6e.
2022-02-28 20:27:11 -06:00