Commit Graph

9913 Commits

Author SHA1 Message Date
evazion
0d835983ce reports: fix error when report is empty.
Fix an exception when a report is empty, for example when performing a
tag search that returns no results:

* https://betabooru.donmai.us/reports/posts?search[group]=uploader&search[tags]=does_not_exist
2022-10-23 21:55:06 -05:00
evazion
5565c753d0 reports: fix exception when using period option and filtering by association.
Fix an exception in reports like this:

* https://betabooru.donmai.us/reports/posts?search[period]=day&search[uploader][name]=evazion

Caused by the `search` method doing a left join instead of a subquery when filtering by a belongs to association.
2022-10-23 21:55:06 -05:00
evazion
bbfea9aecb Update Terms of Service.
Update the copyright section of the Terms of Service to link to the new
DMCA page and to list the same requirements as the DMCA page.
2022-10-23 21:55:06 -05:00
N. Oname
61112bc9a1 Merge pull request #5285 from nonamethanks/tests
Rewrite the tests for various source strategies
2022-10-23 18:05:54 +02:00
evazion
203067b5ed reports: add non-timeseries charts.
Add bar charts for non-timeseries data. For example, a bar chart of the
top 10 uploaders overall in the last month, rather than a timeseries
chart of the number of uploads per day for the last month.
2022-10-23 04:42:51 -05:00
evazion
9ca76dd3be reports: add more reportable tables. 2022-10-22 04:05:10 -05:00
evazion
f73d2e3956 reports: add ability to group reports by column.
Add ability to group reports by various columns. For example, you can see
the posts by the top 10 uploaders over time, or posts grouped by rating
over time.
2022-10-22 04:05:10 -05:00
evazion
f092bc28ee posts: fixup N+1 query issue in 57dc0cbec.
Using `uploader` triggered extra SQL queries to load the uploader of
every post on the post index page.
2022-10-21 01:10:59 -05:00
evazion
0bd749c306 reports: increase database timeout; add rate limits.
Increase the database timeout to 10 seconds when generating reports.
Generating reports tends to be slow, especially for things like graphing
posts over time since the beginning of Danbooru.

Does not apply to anonymous users. Users must have an account to get
higher timeouts so that we can identify users scraping reports too hard.

Also add a rate limit of 1 report per 3 seconds to limit abuse.
2022-10-21 01:04:30 -05:00
evazion
848f47ed96 Merge pull request #5302 from nonamethanks/tag-deprecation-aliases
Tags: remove aliases when deprecating a tag
2022-10-21 01:02:39 -05:00
evazion
dcf7b893de Merge pull request #5301 from nonamethanks/allow-users-to-see-own-uploads
Posts: show levelblocked posts to their uploader
2022-10-21 01:02:23 -05:00
evazion
79d9ca3215 Merge pull request #5300 from nonamethanks/default-colored
Users: display colored usernames by default
2022-10-21 01:01:08 -05:00
evazion
7646521d0f Add basic tables and graphs for various tables.
Add basic tables and graphs for viewing things like uploads over time, new users
over time, comments over time, etc. Located at https://betabooru.donmai.us/reports.

The graphing uses Apache ECharts: https://echarts.apache.org/en/index.html.
2022-10-20 05:20:22 -05:00
nonamethanks
7ebf0002cf Tags: remove aliases when deprecating a tag 2022-10-19 18:36:06 +02:00
nonamethanks
57dc0cbeca Posts: show levelblocked posts to their uploader 2022-10-19 18:07:43 +02:00
nonamethanks
cadb657c5e Users: display colored usernames by default
The previous behavior can be replicated by putting the following in the
custom css:

    .user { color: var(--link-color) !important }
2022-10-19 16:52:17 +02:00
evazion
412b7f2727 http: split requests into internal and external requests.
Split requests made by Danbooru::Http into either internal or external
requests. Internal requests are API calls to internal services run by
Danbooru. External requests are requests to external websites, for
example fetching sources or downloading files. External requests may use
a HTTP proxy if one is configured. Internal requests don't.

Fixes a few source extractors not using the HTTP proxy for certain API calls.
2022-10-19 01:49:28 -05:00
evazion
4001701d18 js: drop typopro-web package.
Drop the typopro-web Javascript package that we used to import the Comic
Relief and Kalam fonts. Include these fonts directly in our own repo
instead. This drops a bunch of unnecessary Javascript dependencies.
Fixes Dependabot alerts about security issues in random Javascript
dependencies pulled in by the typopro-web package.
2022-10-18 20:13:43 -05:00
evazion
6a0e553a84 Remove CurrentUser.country.
This was Cloudflare-specific and it was only used for selecting the
currency during user upgrades.
2022-10-18 18:14:01 -05:00
evazion
873c67db58 emails: disallow names ending with a period.
Update email validation rules to disallow the percent character (e.g.
`foo%bar@gmail.com`) and names ending with a period (e.g. `foo.@gmail.com`).
Names ending with a period are invalid according to the RFCs and cause
`Mail::Address.new` to raise an exception.

The percent character is technically legal, but only one email used it
and it was probably a typo.
2022-10-17 22:13:19 -05:00
evazion
e31977ac29 emails: move EmailValidator into Danbooru::EmailAddress. 2022-10-17 22:13:19 -05:00
evazion
9ea2c34f17 emails: add more typo correction rules for Gmail. 2022-10-17 22:13:19 -05:00
nonamethanks
72528bdcb1 Twitter: rewrite tests
Also add partial support for profile banner images.
2022-10-17 18:53:11 +02:00
evazion
78fa652646 media assets: make file storage paths and URLs configurable.
Add config options to customize where uploads are stored, and how image URLs are generated.

* Add `media_asset_file_path` option to customize where uploads are stored.
* Add `media_asset_file_url` option to customize how image URLs are generated.
* Remove the `enable_seo_post_urls` config option. The `media_asset_file_url` option
  should be used instead to include the tags in the image URL.
2022-10-16 22:36:52 -05:00
evazion
16e74650e8 media assets: include file URLs in /media_assets.json API.
Include information about the asset's variants (sample images) in the /media_assets.json API:

    {
      "id": 6410907,
      "created_at": "2022-07-31T15:44:34.522-04:00",
      "updated_at": "2022-07-31T15:44:38.002-04:00",
      "md5": "19a2be6a1a8582bb349de9734b7a649a",
      "file_ext": "jpg",
      "file_size": 369029,
      "image_width": 600,
      "image_height": 900,
      "duration": null,
      "status": "active",
      "file_key": "R4DBCxBID",
      "is_public": true,
      "variants": [
         {
           "variant": "preview",
           "url": "https://cdn.donmai.us/preview/19/a2/19a2be6a1a8582bb349de9734b7a649a.jpg",
           "width": 100,
           "height": 150,
           "file_ext": "jpg"
         },
         {
           "variant": "180x180",
           "url": "https://cdn.donmai.us/180x180/19/a2/19a2be6a1a8582bb349de9734b7a649a.jpg",
           "width": 120,
           "height": 180,
           "file_ext": "jpg"
         },
         {
           "variant": "360x360",
           "url": "https://cdn.donmai.us/360x360/19/a2/19a2be6a1a8582bb349de9734b7a649a.jpg",
           "width": 240,
           "height": 360,
           "file_ext": "jpg"
         },
         {
           "variant": "720x720",
           "url": "https://cdn.donmai.us/720x720/19/a2/19a2be6a1a8582bb349de9734b7a649a.webp",
           "width": 480,
           "height": 720,
           "file_ext": "webp"
         },
         {
           "variant": "original",
           "url": "https://cdn.donmai.us/original/19/a2/19a2be6a1a8582bb349de9734b7a649a.jpg",
           "width": 600,
           "height": 900,
           "file_ext": "jpg"
         }
      ]
    }
2022-10-16 17:28:23 -05:00
evazion
3b0e94040f posts: fix placeholder thumbnail for Flash files.
* Replace the "Download" placeholder thumbnail for Flash files with a
  new placeholder that specifically says it's a Flash file.
* Fix a bug where the Flash placeholder thumbnail was too small when
  using larger thumbnail sizes.
* Fix it so that media assets don't falsely consider Flash files to have
  thumbnails. This could potentially cause errors if someone tried to
  expunge, replace, or regenerate a Flash post.
2022-10-16 16:46:18 -05:00
evazion
d4492e3c22 related tags: fix nil exception. 2022-10-15 00:20:44 -05:00
evazion
f039d9598a Fix #5287: ActionController::UrlGenerationError in /related_tag.js 2022-10-14 22:54:05 -05:00
evazion
185c8bac82 api: return error if login or api_key params are given but blank.
* Make it an error to supply empty API credentials, like this:
  `https://danbooru.donmai.us/posts.json?login=&api_key=`. Some clients
  did this for some reason.

* Make it so that the `login` and `api_key` params are only allowed as
  URL params, not as POST or PUT body params. Allowing them as body
  params could interfere with the `PUT /api_keys/:id` endpoint, which
  takes an `api_key` param.
2022-10-14 21:27:53 -05:00
evazion
b24e8ae2a7 Fix #5289: NoMethodError in /?search=. 2022-10-14 20:29:29 -05:00
evazion
9c48953e6f Fix #5293: NoMethodError in sitemap when relation is empty. 2022-10-14 19:52:13 -05:00
evazion
2c3a254359 posts: choose better images for opengraph previews.
When choosing the Open Graph image (the preview image shown when a
Danbooru link is posted on Discord or social media), choose the safest
image with the highest score, rather than the image with the highest
favcount.
2022-10-14 19:01:28 -05:00
evazion
edc7e52353 emails: automatically fix typos in email addresses.
Try to automatically fix various kind of typos and common mistakes in
email addresses when a user creates a new account. It's common for users
to signup with addresses like `name@gmai.com`, which leads to bounces
when we try to send the welcome email.
2022-10-14 18:49:33 -05:00
evazion
4dc1a109c5 media metadata: fix exception in /media_metadata.json?only=media_asset.
Fix NameError exception caused by the MediaMetadataController trying to lookup "MediaMetadatum" as the model name.
2022-10-14 15:59:50 -05:00
evazion
93887b00b8 Fix #5294: pixiv_ugoira_frame_data alternative
Allow including the media metadata with the post like this:

* https://danbooru.donmai.us/posts.json?tags=ugoira&only=id,file_url,media_metadata
2022-10-14 15:00:20 -05:00
evazion
4b5d4e5c20 thumbnails: remove crossorigin="anonymous" html attribute.
Possible fix for images not loading when Cloudflare Managed Challenges are enabled.
`crossorigin="anonymous"` prevents the `cf_clearance` cookie from being passed (maybe?).
2022-10-14 02:11:39 -05:00
evazion
115521906c tumblr: fix failure to upload new Tumblr URLs containing the post title.
Fix failure to upload Tumblr URLs of this form:

* https://www.tumblr.com/munespice/683613396085719040/saur-family
2022-10-13 21:11:07 -05:00
evazion
268ec9118a tumblr: fix failure to upload certain video posts.
Fix failure to upload Tumblr video posts that contained a video URL of this form:

* https://va.media.tumblr.com/tumblr_rjoh0hR8Xe1teimlz_720.mp4
2022-10-13 21:10:29 -05:00
evazion
a07234121d tumblr: fixup for parsing www.tumblr.com/name URLs. 2022-10-13 00:26:16 -05:00
evazion
2e7b3cd80b tumblr: normalize https://www.tumblr.com/name artist URLs. 2022-10-12 23:55:17 -05:00
evazion
eb8f98e4a6 artists: normalize foriio.com artist URLs.
Normalize `https://fori.io/comori22` to `https://www.foriio.com/comori22` in artist entries.
2022-10-12 23:46:50 -05:00
evazion
0831b8dc9a Fix #5292: post edit tag_string - unquoted metatag value trailing escapes aren't parsed correctly when last term 2022-10-12 19:33:19 -05:00
evazion
49d0842f00 dmcas: dmail copy of complaint to owner user. 2022-10-12 03:34:05 -05:00
evazion
879243d8f1 dmcas: update wording on form. 2022-10-11 19:30:48 -05:00
evazion
395a3e45f6 css: add more utility classes. 2022-10-11 17:31:19 -05:00
evazion
24bc6aa949 Add DMCA complaint form.
Add a form for submitting DMCA complaints. The complaint is emailed to
the site owner, and a confirmation email is sent to the submitter.
2022-10-11 15:45:47 -05:00
evazion
8fbc6d1d3a gelbooru: fix exception in md5-based post urls.
Fix exception when trying to get the image URL for sources like
https://gelbooru.com/index.php?page=post&s=list&md5=04f2767c64593c3030ce74ecc2528704.
2022-10-11 01:31:49 -05:00
evazion
f05268df7f sources: add Gelbooru support.
Add support for uploading posts from Gelbooru. Note that the translated
tags will include both the Gelbooru tags and the tags from the Gelbooru
post's source. The commentary and artist information will also be taken
from the Gelbooru post's source. The source of the Danbooru post however
will be left as the Gelbooru post itself, not as the Gelbooru post's source.
2022-10-11 00:06:45 -05:00
evazion
c2adf279ee ugoira: remove the PixivUgoiraFrameData model.
Remove the last remaining uses of the PixivUgoiraFrameData model. As of
32bfb8407, Ugoira frame data is now stored in the MediaMetadata model,
under the `Ugoira:FrameDelays` EXIF field.

The pixiv_ugoira_frame_data table still exists, but it can be removed
after this commit is deployed.

Fixes #5264: Error when replacing with ugoira.
2022-10-10 18:21:30 -05:00
evazion
a23c02d0cb posts: include id attribute in API. 2022-10-10 06:28:22 -05:00