Commit Graph

11886 Commits

Author SHA1 Message Date
evazion
7bfdf5cfbe Merge pull request #5303 from nonamethanks/fix-structure
Update structure.sql to the one generated by db:migrate
2022-10-26 03:57:05 -05:00
evazion
7e99be0b2e searchable: better fix for #5312.
Simplify fix for #5312. Fixes an exception when searching `has_one`
associations like `/wiki_pages?search[artist][is_banned]=true`.
2022-10-25 23:30:57 -05:00
evazion
9e2aff874f tests: fix strategy_should_work to not perform API calls outside of tests.
Fix strategy_should_work to not perform API calls outside of `should`
blocks. This could cause the whole test suite to crash if a source test
raised an unexpected exception.
2022-10-25 22:41:36 -05:00
evazion
acea0d5553 Fix #5065: .webp images upload support
Add ability to upload .webp images.

Animated WebP images aren't supported. This is because they aren't
supported by FFmpeg yet[1], so generating thumbnails and samples for
them would be more complicated than for other formats.

[1]: https://trac.ffmpeg.org/ticket/4907
2022-10-25 22:41:36 -05:00
evazion
df0e9bc4a7 uploads: fix it being possible to upload .mkv files as .webm.
Fix it being possible to upload arbitrary .mkv files and have them
be treated as .webm. This was possible because WebM uses the Matroska
container format, and we only checked for the Matroska header, not that
the file was actually a WebM.

There were only 6 such files in production:

* https://danbooru.donmai.us/posts?tags=exif:Matroska:DocType=matroska
* https://danbooru.donmai.us/posts/5522036
* https://danbooru.donmai.us/posts/4743498
* https://danbooru.donmai.us/posts/3925427
* https://danbooru.donmai.us/posts/3147897
* https://danbooru.donmai.us/posts/2965862
* https://danbooru.donmai.us/posts/2430436

These videos are playable in Chrome, but not in Firefox, since Firefox
doesn't support .mkv files (it supports some, depending on which codecs
are used, but not .mkv files in general).
2022-10-25 19:32:31 -05:00
evazion
6413b9abcd Fix #5312: StatementInvalid Error When Querying the media_assets Attribute of Uploads 2022-10-25 16:48:16 -05:00
evazion
c96d60a840 uploads: add support for uploading .avif files.
Features of AVIF include:

* Lossless and lossy compression.
* High dynamic range (HDR) images
* Wide color gamut images (i.e. 10- and 12-bit color depths)
* Transparency (through alpha planes).
* Animations (with an optional cover image).
* Auxiliary image sequences, where the file contains a single primary
  image and a short secondary video, like Apple's Live Photos.
* Metadata rotation, mirroring, and cropping.

The AVIF format is still relatively new and some of these features aren't well
supported by browsers or other software:

* Animated AVIFs aren't supported by Firefox or by libvips.
* HDR images aren't supported by Firefox.
* Rotated, mirrored, and cropped AVIFs aren't supported by Firefox or Chrome.
* Image grids, where the file contains multiple images that are tiled
  together into one big image, aren't supported by Firefox.
* AVIF as a whole has only been supported for a year or two by Chrome
  and Firefox, and less than a year by Safari.

For these reasons, only basic AVIFs that don't use animation, rotation,
cropping, or image grids can be uploaded.
2022-10-25 03:29:58 -05:00
evazion
420ff2f2f5 docker: update base image dependencies.
* Update base Ubuntu image from 22.04 to 22.10.
* Update FFmpeg from 4.4.1 to 5.1.2
* Update MozJPEG from 4.0.3 to 4.1.1.
* Update ExifTool from 12.30 to 12.42.

The FFmpeg upgrade is necessary for .avif file support. Older versions
of ffprobe don't handle .avif files correctly. It either returns an
error, or in certain cases it can interpret a static .avif image as a
video (since .avif files are basically .mp4 files in disguise).
2022-10-24 20:03:37 -05:00
evazion
5b19e544cf Fix #5310: "Details" on disapproval dropdown leads to a broken url. 2022-10-24 01:58:28 -05:00
evazion
0d835983ce reports: fix error when report is empty.
Fix an exception when a report is empty, for example when performing a
tag search that returns no results:

* https://betabooru.donmai.us/reports/posts?search[group]=uploader&search[tags]=does_not_exist
2022-10-23 21:55:06 -05:00
evazion
5565c753d0 reports: fix exception when using period option and filtering by association.
Fix an exception in reports like this:

* https://betabooru.donmai.us/reports/posts?search[period]=day&search[uploader][name]=evazion

Caused by the `search` method doing a left join instead of a subquery when filtering by a belongs to association.
2022-10-23 21:55:06 -05:00
evazion
bbfea9aecb Update Terms of Service.
Update the copyright section of the Terms of Service to link to the new
DMCA page and to list the same requirements as the DMCA page.
2022-10-23 21:55:06 -05:00
N. Oname
61112bc9a1 Merge pull request #5285 from nonamethanks/tests
Rewrite the tests for various source strategies
2022-10-23 18:05:54 +02:00
evazion
203067b5ed reports: add non-timeseries charts.
Add bar charts for non-timeseries data. For example, a bar chart of the
top 10 uploaders overall in the last month, rather than a timeseries
chart of the number of uploads per day for the last month.
2022-10-23 04:42:51 -05:00
nonamethanks
f5a3a7da9f Stash: rewrite tests 2022-10-22 20:09:23 +02:00
nonamethanks
ec293eec2d Plurk: rewrite tests 2022-10-22 20:04:11 +02:00
evazion
9ca76dd3be reports: add more reportable tables. 2022-10-22 04:05:10 -05:00
evazion
f73d2e3956 reports: add ability to group reports by column.
Add ability to group reports by various columns. For example, you can see
the posts by the top 10 uploaders over time, or posts grouped by rating
over time.
2022-10-22 04:05:10 -05:00
nonamethanks
d1b6f30517 Docker compose: add project name to the file.
When docker compose creates the various services, it names the
containers like <projectname>-<servicename>-<id>. If
COMPOSE_PROJECT_NAME is not set and there's no name property
at the root level of the yaml, the default project name is the
current directory. This breaks the deployment from directories
with unusable names, such as C:\.

Fixes #5304.

This should not break existing installations that are pulling from
master, as long as the base folder is danbooru/ (default for git clone).
2022-10-21 18:24:30 +02:00
nonamethanks
79136d92d0 Update structure.sql to the one generated by db:migrate 2022-10-21 11:22:41 +02:00
nonamethanks
1b11321161 Pixiv Sketch: rewrite tests 2022-10-21 11:09:10 +02:00
evazion
f092bc28ee posts: fixup N+1 query issue in 57dc0cbec.
Using `uploader` triggered extra SQL queries to load the uploader of
every post on the post index page.
2022-10-21 01:10:59 -05:00
evazion
0bd749c306 reports: increase database timeout; add rate limits.
Increase the database timeout to 10 seconds when generating reports.
Generating reports tends to be slow, especially for things like graphing
posts over time since the beginning of Danbooru.

Does not apply to anonymous users. Users must have an account to get
higher timeouts so that we can identify users scraping reports too hard.

Also add a rate limit of 1 report per 3 seconds to limit abuse.
2022-10-21 01:04:30 -05:00
evazion
848f47ed96 Merge pull request #5302 from nonamethanks/tag-deprecation-aliases
Tags: remove aliases when deprecating a tag
2022-10-21 01:02:39 -05:00
evazion
dcf7b893de Merge pull request #5301 from nonamethanks/allow-users-to-see-own-uploads
Posts: show levelblocked posts to their uploader
2022-10-21 01:02:23 -05:00
evazion
4c313cb045 Merge pull request #5298 from nonamethanks/fix-docker-compose
Docker compose: add comments and use templating
2022-10-21 01:01:26 -05:00
evazion
79d9ca3215 Merge pull request #5300 from nonamethanks/default-colored
Users: display colored usernames by default
2022-10-21 01:01:08 -05:00
evazion
7646521d0f Add basic tables and graphs for various tables.
Add basic tables and graphs for viewing things like uploads over time, new users
over time, comments over time, etc. Located at https://betabooru.donmai.us/reports.

The graphing uses Apache ECharts: https://echarts.apache.org/en/index.html.
2022-10-20 05:20:22 -05:00
nonamethanks
ec05d21091 Docker compose: add comments and use templating 2022-10-19 21:44:03 +02:00
nonamethanks
7ebf0002cf Tags: remove aliases when deprecating a tag 2022-10-19 18:36:06 +02:00
nonamethanks
57dc0cbeca Posts: show levelblocked posts to their uploader 2022-10-19 18:07:43 +02:00
nonamethanks
cadb657c5e Users: display colored usernames by default
The previous behavior can be replicated by putting the following in the
custom css:

    .user { color: var(--link-color) !important }
2022-10-19 16:52:17 +02:00
evazion
412b7f2727 http: split requests into internal and external requests.
Split requests made by Danbooru::Http into either internal or external
requests. Internal requests are API calls to internal services run by
Danbooru. External requests are requests to external websites, for
example fetching sources or downloading files. External requests may use
a HTTP proxy if one is configured. Internal requests don't.

Fixes a few source extractors not using the HTTP proxy for certain API calls.
2022-10-19 01:49:28 -05:00
evazion
4001701d18 js: drop typopro-web package.
Drop the typopro-web Javascript package that we used to import the Comic
Relief and Kalam fonts. Include these fonts directly in our own repo
instead. This drops a bunch of unnecessary Javascript dependencies.
Fixes Dependabot alerts about security issues in random Javascript
dependencies pulled in by the typopro-web package.
2022-10-18 20:13:43 -05:00
evazion
2989ba1854 Update Ruby gems and Yarn packages. 2022-10-18 19:17:26 -05:00
evazion
6a0e553a84 Remove CurrentUser.country.
This was Cloudflare-specific and it was only used for selecting the
currency during user upgrades.
2022-10-18 18:14:01 -05:00
evazion
873c67db58 emails: disallow names ending with a period.
Update email validation rules to disallow the percent character (e.g.
`foo%bar@gmail.com`) and names ending with a period (e.g. `foo.@gmail.com`).
Names ending with a period are invalid according to the RFCs and cause
`Mail::Address.new` to raise an exception.

The percent character is technically legal, but only one email used it
and it was probably a typo.
2022-10-17 22:13:19 -05:00
evazion
e31977ac29 emails: move EmailValidator into Danbooru::EmailAddress. 2022-10-17 22:13:19 -05:00
evazion
9ea2c34f17 emails: add more typo correction rules for Gmail. 2022-10-17 22:13:19 -05:00
nonamethanks
72528bdcb1 Twitter: rewrite tests
Also add partial support for profile banner images.
2022-10-17 18:53:11 +02:00
evazion
78fa652646 media assets: make file storage paths and URLs configurable.
Add config options to customize where uploads are stored, and how image URLs are generated.

* Add `media_asset_file_path` option to customize where uploads are stored.
* Add `media_asset_file_url` option to customize how image URLs are generated.
* Remove the `enable_seo_post_urls` config option. The `media_asset_file_url` option
  should be used instead to include the tags in the image URL.
2022-10-16 22:36:52 -05:00
evazion
16e74650e8 media assets: include file URLs in /media_assets.json API.
Include information about the asset's variants (sample images) in the /media_assets.json API:

    {
      "id": 6410907,
      "created_at": "2022-07-31T15:44:34.522-04:00",
      "updated_at": "2022-07-31T15:44:38.002-04:00",
      "md5": "19a2be6a1a8582bb349de9734b7a649a",
      "file_ext": "jpg",
      "file_size": 369029,
      "image_width": 600,
      "image_height": 900,
      "duration": null,
      "status": "active",
      "file_key": "R4DBCxBID",
      "is_public": true,
      "variants": [
         {
           "variant": "preview",
           "url": "https://cdn.donmai.us/preview/19/a2/19a2be6a1a8582bb349de9734b7a649a.jpg",
           "width": 100,
           "height": 150,
           "file_ext": "jpg"
         },
         {
           "variant": "180x180",
           "url": "https://cdn.donmai.us/180x180/19/a2/19a2be6a1a8582bb349de9734b7a649a.jpg",
           "width": 120,
           "height": 180,
           "file_ext": "jpg"
         },
         {
           "variant": "360x360",
           "url": "https://cdn.donmai.us/360x360/19/a2/19a2be6a1a8582bb349de9734b7a649a.jpg",
           "width": 240,
           "height": 360,
           "file_ext": "jpg"
         },
         {
           "variant": "720x720",
           "url": "https://cdn.donmai.us/720x720/19/a2/19a2be6a1a8582bb349de9734b7a649a.webp",
           "width": 480,
           "height": 720,
           "file_ext": "webp"
         },
         {
           "variant": "original",
           "url": "https://cdn.donmai.us/original/19/a2/19a2be6a1a8582bb349de9734b7a649a.jpg",
           "width": 600,
           "height": 900,
           "file_ext": "jpg"
         }
      ]
    }
2022-10-16 17:28:23 -05:00
evazion
3b0e94040f posts: fix placeholder thumbnail for Flash files.
* Replace the "Download" placeholder thumbnail for Flash files with a
  new placeholder that specifically says it's a Flash file.
* Fix a bug where the Flash placeholder thumbnail was too small when
  using larger thumbnail sizes.
* Fix it so that media assets don't falsely consider Flash files to have
  thumbnails. This could potentially cause errors if someone tried to
  expunge, replace, or regenerate a Flash post.
2022-10-16 16:46:18 -05:00
evazion
d4492e3c22 related tags: fix nil exception. 2022-10-15 00:20:44 -05:00
evazion
f6516e0e37 emails: add script to fix typo'd emails.
Add fix script to fix emails containing typos, such as `name@gamil.com`.
2022-10-14 23:28:23 -05:00
evazion
f039d9598a Fix #5287: ActionController::UrlGenerationError in /related_tag.js 2022-10-14 22:54:05 -05:00
evazion
185c8bac82 api: return error if login or api_key params are given but blank.
* Make it an error to supply empty API credentials, like this:
  `https://danbooru.donmai.us/posts.json?login=&api_key=`. Some clients
  did this for some reason.

* Make it so that the `login` and `api_key` params are only allowed as
  URL params, not as POST or PUT body params. Allowing them as body
  params could interfere with the `PUT /api_keys/:id` endpoint, which
  takes an `api_key` param.
2022-10-14 21:27:53 -05:00
evazion
b24e8ae2a7 Fix #5289: NoMethodError in /?search=. 2022-10-14 20:29:29 -05:00
evazion
9c48953e6f Fix #5293: NoMethodError in sitemap when relation is empty. 2022-10-14 19:52:13 -05:00
evazion
2c3a254359 posts: choose better images for opengraph previews.
When choosing the Open Graph image (the preview image shown when a
Danbooru link is posted on Discord or social media), choose the safest
image with the highest score, rather than the image with the highest
favcount.
2022-10-14 19:01:28 -05:00