Commit Graph

9921 Commits

Author SHA1 Message Date
evazion
81d9cf4b96 uploads: add .avif and .webp extensions to file picker dialog. 2022-10-26 04:15:37 -05:00
evazion
9c811611c6 media assets: add full variant for .avif and .webp files.
Add a JPEG conversion for .avif and .webp files. The `full` variant is
the .avif or .webp file converted to JPEG format, with the same
resolution as the original file (full resolution).

Known bug: When converting an HDR .avif file to .jpeg, the resulting
image is too bright compared to the original image as rendered by
Firefox or Chrome.
2022-10-26 04:09:59 -05:00
evazion
7e99be0b2e searchable: better fix for #5312.
Simplify fix for #5312. Fixes an exception when searching `has_one`
associations like `/wiki_pages?search[artist][is_banned]=true`.
2022-10-25 23:30:57 -05:00
evazion
acea0d5553 Fix #5065: .webp images upload support
Add ability to upload .webp images.

Animated WebP images aren't supported. This is because they aren't
supported by FFmpeg yet[1], so generating thumbnails and samples for
them would be more complicated than for other formats.

[1]: https://trac.ffmpeg.org/ticket/4907
2022-10-25 22:41:36 -05:00
evazion
df0e9bc4a7 uploads: fix it being possible to upload .mkv files as .webm.
Fix it being possible to upload arbitrary .mkv files and have them
be treated as .webm. This was possible because WebM uses the Matroska
container format, and we only checked for the Matroska header, not that
the file was actually a WebM.

There were only 6 such files in production:

* https://danbooru.donmai.us/posts?tags=exif:Matroska:DocType=matroska
* https://danbooru.donmai.us/posts/5522036
* https://danbooru.donmai.us/posts/4743498
* https://danbooru.donmai.us/posts/3925427
* https://danbooru.donmai.us/posts/3147897
* https://danbooru.donmai.us/posts/2965862
* https://danbooru.donmai.us/posts/2430436

These videos are playable in Chrome, but not in Firefox, since Firefox
doesn't support .mkv files (it supports some, depending on which codecs
are used, but not .mkv files in general).
2022-10-25 19:32:31 -05:00
evazion
6413b9abcd Fix #5312: StatementInvalid Error When Querying the media_assets Attribute of Uploads 2022-10-25 16:48:16 -05:00
evazion
c96d60a840 uploads: add support for uploading .avif files.
Features of AVIF include:

* Lossless and lossy compression.
* High dynamic range (HDR) images
* Wide color gamut images (i.e. 10- and 12-bit color depths)
* Transparency (through alpha planes).
* Animations (with an optional cover image).
* Auxiliary image sequences, where the file contains a single primary
  image and a short secondary video, like Apple's Live Photos.
* Metadata rotation, mirroring, and cropping.

The AVIF format is still relatively new and some of these features aren't well
supported by browsers or other software:

* Animated AVIFs aren't supported by Firefox or by libvips.
* HDR images aren't supported by Firefox.
* Rotated, mirrored, and cropped AVIFs aren't supported by Firefox or Chrome.
* Image grids, where the file contains multiple images that are tiled
  together into one big image, aren't supported by Firefox.
* AVIF as a whole has only been supported for a year or two by Chrome
  and Firefox, and less than a year by Safari.

For these reasons, only basic AVIFs that don't use animation, rotation,
cropping, or image grids can be uploaded.
2022-10-25 03:29:58 -05:00
evazion
5b19e544cf Fix #5310: "Details" on disapproval dropdown leads to a broken url. 2022-10-24 01:58:28 -05:00
evazion
0d835983ce reports: fix error when report is empty.
Fix an exception when a report is empty, for example when performing a
tag search that returns no results:

* https://betabooru.donmai.us/reports/posts?search[group]=uploader&search[tags]=does_not_exist
2022-10-23 21:55:06 -05:00
evazion
5565c753d0 reports: fix exception when using period option and filtering by association.
Fix an exception in reports like this:

* https://betabooru.donmai.us/reports/posts?search[period]=day&search[uploader][name]=evazion

Caused by the `search` method doing a left join instead of a subquery when filtering by a belongs to association.
2022-10-23 21:55:06 -05:00
evazion
bbfea9aecb Update Terms of Service.
Update the copyright section of the Terms of Service to link to the new
DMCA page and to list the same requirements as the DMCA page.
2022-10-23 21:55:06 -05:00
N. Oname
61112bc9a1 Merge pull request #5285 from nonamethanks/tests
Rewrite the tests for various source strategies
2022-10-23 18:05:54 +02:00
evazion
203067b5ed reports: add non-timeseries charts.
Add bar charts for non-timeseries data. For example, a bar chart of the
top 10 uploaders overall in the last month, rather than a timeseries
chart of the number of uploads per day for the last month.
2022-10-23 04:42:51 -05:00
evazion
9ca76dd3be reports: add more reportable tables. 2022-10-22 04:05:10 -05:00
evazion
f73d2e3956 reports: add ability to group reports by column.
Add ability to group reports by various columns. For example, you can see
the posts by the top 10 uploaders over time, or posts grouped by rating
over time.
2022-10-22 04:05:10 -05:00
evazion
f092bc28ee posts: fixup N+1 query issue in 57dc0cbec.
Using `uploader` triggered extra SQL queries to load the uploader of
every post on the post index page.
2022-10-21 01:10:59 -05:00
evazion
0bd749c306 reports: increase database timeout; add rate limits.
Increase the database timeout to 10 seconds when generating reports.
Generating reports tends to be slow, especially for things like graphing
posts over time since the beginning of Danbooru.

Does not apply to anonymous users. Users must have an account to get
higher timeouts so that we can identify users scraping reports too hard.

Also add a rate limit of 1 report per 3 seconds to limit abuse.
2022-10-21 01:04:30 -05:00
evazion
848f47ed96 Merge pull request #5302 from nonamethanks/tag-deprecation-aliases
Tags: remove aliases when deprecating a tag
2022-10-21 01:02:39 -05:00
evazion
dcf7b893de Merge pull request #5301 from nonamethanks/allow-users-to-see-own-uploads
Posts: show levelblocked posts to their uploader
2022-10-21 01:02:23 -05:00
evazion
79d9ca3215 Merge pull request #5300 from nonamethanks/default-colored
Users: display colored usernames by default
2022-10-21 01:01:08 -05:00
evazion
7646521d0f Add basic tables and graphs for various tables.
Add basic tables and graphs for viewing things like uploads over time, new users
over time, comments over time, etc. Located at https://betabooru.donmai.us/reports.

The graphing uses Apache ECharts: https://echarts.apache.org/en/index.html.
2022-10-20 05:20:22 -05:00
nonamethanks
7ebf0002cf Tags: remove aliases when deprecating a tag 2022-10-19 18:36:06 +02:00
nonamethanks
57dc0cbeca Posts: show levelblocked posts to their uploader 2022-10-19 18:07:43 +02:00
nonamethanks
cadb657c5e Users: display colored usernames by default
The previous behavior can be replicated by putting the following in the
custom css:

    .user { color: var(--link-color) !important }
2022-10-19 16:52:17 +02:00
evazion
412b7f2727 http: split requests into internal and external requests.
Split requests made by Danbooru::Http into either internal or external
requests. Internal requests are API calls to internal services run by
Danbooru. External requests are requests to external websites, for
example fetching sources or downloading files. External requests may use
a HTTP proxy if one is configured. Internal requests don't.

Fixes a few source extractors not using the HTTP proxy for certain API calls.
2022-10-19 01:49:28 -05:00
evazion
4001701d18 js: drop typopro-web package.
Drop the typopro-web Javascript package that we used to import the Comic
Relief and Kalam fonts. Include these fonts directly in our own repo
instead. This drops a bunch of unnecessary Javascript dependencies.
Fixes Dependabot alerts about security issues in random Javascript
dependencies pulled in by the typopro-web package.
2022-10-18 20:13:43 -05:00
evazion
6a0e553a84 Remove CurrentUser.country.
This was Cloudflare-specific and it was only used for selecting the
currency during user upgrades.
2022-10-18 18:14:01 -05:00
evazion
873c67db58 emails: disallow names ending with a period.
Update email validation rules to disallow the percent character (e.g.
`foo%bar@gmail.com`) and names ending with a period (e.g. `foo.@gmail.com`).
Names ending with a period are invalid according to the RFCs and cause
`Mail::Address.new` to raise an exception.

The percent character is technically legal, but only one email used it
and it was probably a typo.
2022-10-17 22:13:19 -05:00
evazion
e31977ac29 emails: move EmailValidator into Danbooru::EmailAddress. 2022-10-17 22:13:19 -05:00
evazion
9ea2c34f17 emails: add more typo correction rules for Gmail. 2022-10-17 22:13:19 -05:00
nonamethanks
72528bdcb1 Twitter: rewrite tests
Also add partial support for profile banner images.
2022-10-17 18:53:11 +02:00
evazion
78fa652646 media assets: make file storage paths and URLs configurable.
Add config options to customize where uploads are stored, and how image URLs are generated.

* Add `media_asset_file_path` option to customize where uploads are stored.
* Add `media_asset_file_url` option to customize how image URLs are generated.
* Remove the `enable_seo_post_urls` config option. The `media_asset_file_url` option
  should be used instead to include the tags in the image URL.
2022-10-16 22:36:52 -05:00
evazion
16e74650e8 media assets: include file URLs in /media_assets.json API.
Include information about the asset's variants (sample images) in the /media_assets.json API:

    {
      "id": 6410907,
      "created_at": "2022-07-31T15:44:34.522-04:00",
      "updated_at": "2022-07-31T15:44:38.002-04:00",
      "md5": "19a2be6a1a8582bb349de9734b7a649a",
      "file_ext": "jpg",
      "file_size": 369029,
      "image_width": 600,
      "image_height": 900,
      "duration": null,
      "status": "active",
      "file_key": "R4DBCxBID",
      "is_public": true,
      "variants": [
         {
           "variant": "preview",
           "url": "https://cdn.donmai.us/preview/19/a2/19a2be6a1a8582bb349de9734b7a649a.jpg",
           "width": 100,
           "height": 150,
           "file_ext": "jpg"
         },
         {
           "variant": "180x180",
           "url": "https://cdn.donmai.us/180x180/19/a2/19a2be6a1a8582bb349de9734b7a649a.jpg",
           "width": 120,
           "height": 180,
           "file_ext": "jpg"
         },
         {
           "variant": "360x360",
           "url": "https://cdn.donmai.us/360x360/19/a2/19a2be6a1a8582bb349de9734b7a649a.jpg",
           "width": 240,
           "height": 360,
           "file_ext": "jpg"
         },
         {
           "variant": "720x720",
           "url": "https://cdn.donmai.us/720x720/19/a2/19a2be6a1a8582bb349de9734b7a649a.webp",
           "width": 480,
           "height": 720,
           "file_ext": "webp"
         },
         {
           "variant": "original",
           "url": "https://cdn.donmai.us/original/19/a2/19a2be6a1a8582bb349de9734b7a649a.jpg",
           "width": 600,
           "height": 900,
           "file_ext": "jpg"
         }
      ]
    }
2022-10-16 17:28:23 -05:00
evazion
3b0e94040f posts: fix placeholder thumbnail for Flash files.
* Replace the "Download" placeholder thumbnail for Flash files with a
  new placeholder that specifically says it's a Flash file.
* Fix a bug where the Flash placeholder thumbnail was too small when
  using larger thumbnail sizes.
* Fix it so that media assets don't falsely consider Flash files to have
  thumbnails. This could potentially cause errors if someone tried to
  expunge, replace, or regenerate a Flash post.
2022-10-16 16:46:18 -05:00
evazion
d4492e3c22 related tags: fix nil exception. 2022-10-15 00:20:44 -05:00
evazion
f039d9598a Fix #5287: ActionController::UrlGenerationError in /related_tag.js 2022-10-14 22:54:05 -05:00
evazion
185c8bac82 api: return error if login or api_key params are given but blank.
* Make it an error to supply empty API credentials, like this:
  `https://danbooru.donmai.us/posts.json?login=&api_key=`. Some clients
  did this for some reason.

* Make it so that the `login` and `api_key` params are only allowed as
  URL params, not as POST or PUT body params. Allowing them as body
  params could interfere with the `PUT /api_keys/:id` endpoint, which
  takes an `api_key` param.
2022-10-14 21:27:53 -05:00
evazion
b24e8ae2a7 Fix #5289: NoMethodError in /?search=. 2022-10-14 20:29:29 -05:00
evazion
9c48953e6f Fix #5293: NoMethodError in sitemap when relation is empty. 2022-10-14 19:52:13 -05:00
evazion
2c3a254359 posts: choose better images for opengraph previews.
When choosing the Open Graph image (the preview image shown when a
Danbooru link is posted on Discord or social media), choose the safest
image with the highest score, rather than the image with the highest
favcount.
2022-10-14 19:01:28 -05:00
evazion
edc7e52353 emails: automatically fix typos in email addresses.
Try to automatically fix various kind of typos and common mistakes in
email addresses when a user creates a new account. It's common for users
to signup with addresses like `name@gmai.com`, which leads to bounces
when we try to send the welcome email.
2022-10-14 18:49:33 -05:00
evazion
4dc1a109c5 media metadata: fix exception in /media_metadata.json?only=media_asset.
Fix NameError exception caused by the MediaMetadataController trying to lookup "MediaMetadatum" as the model name.
2022-10-14 15:59:50 -05:00
evazion
93887b00b8 Fix #5294: pixiv_ugoira_frame_data alternative
Allow including the media metadata with the post like this:

* https://danbooru.donmai.us/posts.json?tags=ugoira&only=id,file_url,media_metadata
2022-10-14 15:00:20 -05:00
evazion
4b5d4e5c20 thumbnails: remove crossorigin="anonymous" html attribute.
Possible fix for images not loading when Cloudflare Managed Challenges are enabled.
`crossorigin="anonymous"` prevents the `cf_clearance` cookie from being passed (maybe?).
2022-10-14 02:11:39 -05:00
evazion
115521906c tumblr: fix failure to upload new Tumblr URLs containing the post title.
Fix failure to upload Tumblr URLs of this form:

* https://www.tumblr.com/munespice/683613396085719040/saur-family
2022-10-13 21:11:07 -05:00
evazion
268ec9118a tumblr: fix failure to upload certain video posts.
Fix failure to upload Tumblr video posts that contained a video URL of this form:

* https://va.media.tumblr.com/tumblr_rjoh0hR8Xe1teimlz_720.mp4
2022-10-13 21:10:29 -05:00
evazion
a07234121d tumblr: fixup for parsing www.tumblr.com/name URLs. 2022-10-13 00:26:16 -05:00
evazion
2e7b3cd80b tumblr: normalize https://www.tumblr.com/name artist URLs. 2022-10-12 23:55:17 -05:00
evazion
eb8f98e4a6 artists: normalize foriio.com artist URLs.
Normalize `https://fori.io/comori22` to `https://www.foriio.com/comori22` in artist entries.
2022-10-12 23:46:50 -05:00
evazion
0831b8dc9a Fix #5292: post edit tag_string - unquoted metatag value trailing escapes aren't parsed correctly when last term 2022-10-12 19:33:19 -05:00