Commit Graph

2662 Commits

Author SHA1 Message Date
evazion
9e2aff874f tests: fix strategy_should_work to not perform API calls outside of tests.
Fix strategy_should_work to not perform API calls outside of `should`
blocks. This could cause the whole test suite to crash if a source test
raised an unexpected exception.
2022-10-25 22:41:36 -05:00
evazion
acea0d5553 Fix #5065: .webp images upload support
Add ability to upload .webp images.

Animated WebP images aren't supported. This is because they aren't
supported by FFmpeg yet[1], so generating thumbnails and samples for
them would be more complicated than for other formats.

[1]: https://trac.ffmpeg.org/ticket/4907
2022-10-25 22:41:36 -05:00
evazion
df0e9bc4a7 uploads: fix it being possible to upload .mkv files as .webm.
Fix it being possible to upload arbitrary .mkv files and have them
be treated as .webm. This was possible because WebM uses the Matroska
container format, and we only checked for the Matroska header, not that
the file was actually a WebM.

There were only 6 such files in production:

* https://danbooru.donmai.us/posts?tags=exif:Matroska:DocType=matroska
* https://danbooru.donmai.us/posts/5522036
* https://danbooru.donmai.us/posts/4743498
* https://danbooru.donmai.us/posts/3925427
* https://danbooru.donmai.us/posts/3147897
* https://danbooru.donmai.us/posts/2965862
* https://danbooru.donmai.us/posts/2430436

These videos are playable in Chrome, but not in Firefox, since Firefox
doesn't support .mkv files (it supports some, depending on which codecs
are used, but not .mkv files in general).
2022-10-25 19:32:31 -05:00
evazion
6413b9abcd Fix #5312: StatementInvalid Error When Querying the media_assets Attribute of Uploads 2022-10-25 16:48:16 -05:00
evazion
c96d60a840 uploads: add support for uploading .avif files.
Features of AVIF include:

* Lossless and lossy compression.
* High dynamic range (HDR) images
* Wide color gamut images (i.e. 10- and 12-bit color depths)
* Transparency (through alpha planes).
* Animations (with an optional cover image).
* Auxiliary image sequences, where the file contains a single primary
  image and a short secondary video, like Apple's Live Photos.
* Metadata rotation, mirroring, and cropping.

The AVIF format is still relatively new and some of these features aren't well
supported by browsers or other software:

* Animated AVIFs aren't supported by Firefox or by libvips.
* HDR images aren't supported by Firefox.
* Rotated, mirrored, and cropped AVIFs aren't supported by Firefox or Chrome.
* Image grids, where the file contains multiple images that are tiled
  together into one big image, aren't supported by Firefox.
* AVIF as a whole has only been supported for a year or two by Chrome
  and Firefox, and less than a year by Safari.

For these reasons, only basic AVIFs that don't use animation, rotation,
cropping, or image grids can be uploaded.
2022-10-25 03:29:58 -05:00
evazion
5b19e544cf Fix #5310: "Details" on disapproval dropdown leads to a broken url. 2022-10-24 01:58:28 -05:00
evazion
0d835983ce reports: fix error when report is empty.
Fix an exception when a report is empty, for example when performing a
tag search that returns no results:

* https://betabooru.donmai.us/reports/posts?search[group]=uploader&search[tags]=does_not_exist
2022-10-23 21:55:06 -05:00
evazion
5565c753d0 reports: fix exception when using period option and filtering by association.
Fix an exception in reports like this:

* https://betabooru.donmai.us/reports/posts?search[period]=day&search[uploader][name]=evazion

Caused by the `search` method doing a left join instead of a subquery when filtering by a belongs to association.
2022-10-23 21:55:06 -05:00
N. Oname
61112bc9a1 Merge pull request #5285 from nonamethanks/tests
Rewrite the tests for various source strategies
2022-10-23 18:05:54 +02:00
nonamethanks
f5a3a7da9f Stash: rewrite tests 2022-10-22 20:09:23 +02:00
nonamethanks
ec293eec2d Plurk: rewrite tests 2022-10-22 20:04:11 +02:00
nonamethanks
1b11321161 Pixiv Sketch: rewrite tests 2022-10-21 11:09:10 +02:00
nonamethanks
7ebf0002cf Tags: remove aliases when deprecating a tag 2022-10-19 18:36:06 +02:00
evazion
873c67db58 emails: disallow names ending with a period.
Update email validation rules to disallow the percent character (e.g.
`foo%bar@gmail.com`) and names ending with a period (e.g. `foo.@gmail.com`).
Names ending with a period are invalid according to the RFCs and cause
`Mail::Address.new` to raise an exception.

The percent character is technically legal, but only one email used it
and it was probably a typo.
2022-10-17 22:13:19 -05:00
evazion
e31977ac29 emails: move EmailValidator into Danbooru::EmailAddress. 2022-10-17 22:13:19 -05:00
nonamethanks
72528bdcb1 Twitter: rewrite tests
Also add partial support for profile banner images.
2022-10-17 18:53:11 +02:00
evazion
185c8bac82 api: return error if login or api_key params are given but blank.
* Make it an error to supply empty API credentials, like this:
  `https://danbooru.donmai.us/posts.json?login=&api_key=`. Some clients
  did this for some reason.

* Make it so that the `login` and `api_key` params are only allowed as
  URL params, not as POST or PUT body params. Allowing them as body
  params could interfere with the `PUT /api_keys/:id` endpoint, which
  takes an `api_key` param.
2022-10-14 21:27:53 -05:00
evazion
b24e8ae2a7 Fix #5289: NoMethodError in /?search=. 2022-10-14 20:29:29 -05:00
evazion
9c48953e6f Fix #5293: NoMethodError in sitemap when relation is empty. 2022-10-14 19:52:13 -05:00
evazion
edc7e52353 emails: automatically fix typos in email addresses.
Try to automatically fix various kind of typos and common mistakes in
email addresses when a user creates a new account. It's common for users
to signup with addresses like `name@gmai.com`, which leads to bounces
when we try to send the welcome email.
2022-10-14 18:49:33 -05:00
evazion
4dc1a109c5 media metadata: fix exception in /media_metadata.json?only=media_asset.
Fix NameError exception caused by the MediaMetadataController trying to lookup "MediaMetadatum" as the model name.
2022-10-14 15:59:50 -05:00
evazion
115521906c tumblr: fix failure to upload new Tumblr URLs containing the post title.
Fix failure to upload Tumblr URLs of this form:

* https://www.tumblr.com/munespice/683613396085719040/saur-family
2022-10-13 21:11:07 -05:00
evazion
268ec9118a tumblr: fix failure to upload certain video posts.
Fix failure to upload Tumblr video posts that contained a video URL of this form:

* https://va.media.tumblr.com/tumblr_rjoh0hR8Xe1teimlz_720.mp4
2022-10-13 21:10:29 -05:00
evazion
2e7b3cd80b tumblr: normalize https://www.tumblr.com/name artist URLs. 2022-10-12 23:55:17 -05:00
evazion
0831b8dc9a Fix #5292: post edit tag_string - unquoted metatag value trailing escapes aren't parsed correctly when last term 2022-10-12 19:33:19 -05:00
evazion
49d0842f00 dmcas: dmail copy of complaint to owner user. 2022-10-12 03:34:05 -05:00
evazion
24bc6aa949 Add DMCA complaint form.
Add a form for submitting DMCA complaints. The complaint is emailed to
the site owner, and a confirmation email is sent to the submitter.
2022-10-11 15:45:47 -05:00
evazion
8fbc6d1d3a gelbooru: fix exception in md5-based post urls.
Fix exception when trying to get the image URL for sources like
https://gelbooru.com/index.php?page=post&s=list&md5=04f2767c64593c3030ce74ecc2528704.
2022-10-11 01:31:49 -05:00
evazion
f05268df7f sources: add Gelbooru support.
Add support for uploading posts from Gelbooru. Note that the translated
tags will include both the Gelbooru tags and the tags from the Gelbooru
post's source. The commentary and artist information will also be taken
from the Gelbooru post's source. The source of the Danbooru post however
will be left as the Gelbooru post itself, not as the Gelbooru post's source.
2022-10-11 00:06:45 -05:00
evazion
c2adf279ee ugoira: remove the PixivUgoiraFrameData model.
Remove the last remaining uses of the PixivUgoiraFrameData model. As of
32bfb8407, Ugoira frame data is now stored in the MediaMetadata model,
under the `Ugoira:FrameDelays` EXIF field.

The pixiv_ugoira_frame_data table still exists, but it can be removed
after this commit is deployed.

Fixes #5264: Error when replacing with ugoira.
2022-10-10 18:21:30 -05:00
evazion
01d10a54f8 ugoira: store frame delays in MediaMetadata model.
Store Ugoira frame delays in the MediaMetadata model as a fake EXIF
field instead of in the PixivUgoiraFrameData model. This way we can get
rid of the PixivUgoiraFrameData model completely. This is a step towards
fixing #5264.
2022-10-09 22:25:20 -05:00
nonamethanks
8008b7a5a2 Newgrounds: rewrite tests 2022-10-08 16:23:14 +02:00
nonamethanks
79a9081efa Moebooru: rewrite tests 2022-10-08 16:10:39 +02:00
nonamethanks
f4b14ba23e Mastodon: rewrite tests 2022-10-08 15:55:06 +02:00
nonamethanks
da46b13a64 Hentai-Foundry: rewrite tests 2022-10-07 22:22:24 +02:00
nonamethanks
1dc60feeb7 Fanbox: rewrite tests 2022-10-07 22:04:00 +02:00
nonamethanks
3c8e8ad8d9 Artstation: rewrite tests 2022-10-07 21:37:22 +02:00
evazion
b988711bce tests: fix broken tests. 2022-10-06 23:16:27 -05:00
evazion
99846b7e3d users: allow mods to change the names of other users.
Allow moderators to forcibly change the username of other users. This is
so mods can change abusive or invalid usernames.

* A mod can only change the username of Builder-level users and below.
* The user can't change their own name again until one week has passed.
* A modaction is logged when a mod changes a user's name.
* A dmail is sent to the user notifying them of the change.
* The dmail does not send the user an email notification. This is so we
  don't spam users if their name is changed after they're banned, or if
  they haven't visited the site in a long time.

The rename button is on the user's profile page, and when you hover over
the user's name and open the "..." menu.
2022-10-02 01:32:10 -05:00
nonamethanks
775326dc37 Tumblr: fix crash when uploading image links from custom domains 2022-10-01 00:26:29 +02:00
nonamethanks
1d7caf703c Lofter: support another theme and rewrite tests 2022-09-30 22:04:40 +02:00
evazion
559bf1ae0a modqueue: fix the disapproved: metatag showing posts outside the queue.
Fix a bug where filtering the modqueue by the `disapproved:<reason>` tag
would return posts outside the modqueue.
2022-09-30 02:03:15 -05:00
nonamethanks
d51cc17eaf Nicoseiga: rewrite tests and fix several bugs
* Fixed a bug where manga posts with a single tag would raise an error
* Fixed a bug where dic.nicovideo.jp/oekaki posts weren't uploadable due
  to SSL issues
* Added support for more manga corner cases
2022-09-29 14:37:46 +02:00
evazion
d629c81aa1 Fix #5267: order[custom] no longer works when only a single ID is being searched 2022-09-29 04:36:12 -05:00
evazion
530d8cf762 searchable: fix searching for invalid IP addresses.
Fix an ArgumentError exception when searching for an invalid IP address.

Also allow searching for multiple subnets at once.
2022-09-29 04:36:12 -05:00
evazion
c5b215ffcb searchable: fix searching for polymorphic attributes.
Fix not being able to use the full set of search operators on polymorphic `model_id` and
`model_type` attributes. Before things like `search[model_type]=Post` worked, but
`search[model_type_not_eq]=Post` or other `model_type_*` operators didn't.
2022-09-29 04:36:12 -05:00
evazion
b94cb7d824 emails: include logging information in email headers.
Log the following information in email headers:

* X-Danbooru-User: the user's name and ID.
* X-Danbooru-IP: the user's IP.
* X-Danbooru-Session: the users' session ID.
* X-Danbooru-URL: the page that triggered the email.
* X-Danbooru-Job-Id: the ID of the background job that sent the email.
* X-Danbooru-Enqueued-At: when the email was queued as a background job.
* X-Danbooru-Dmail: for Dmail notifications, the link to the Dmail.
* X-Request-Id: the request ID of the HTTP request that triggered the email.

Also make it so we log an event in the APM when we send an email.
2022-09-29 04:36:11 -05:00
evazion
ed9986def6 emails: fix one-click unsubscription.
Fix it so that emails are (hopefully) able to show the one-click
unsubscribe button in Gmail and other mail providers that support the
List-Unsubscribe header. This way users can unsubscribe instead of
marking emails as spam.

* Add the List-Unsubscribe-Post header.
* Fix the disable email notifications endpoint to support POST as well as DELETE requests.
* Fix the disable email notifications endpoint to disable XSRF protection (we don't need
  users to be logged in because we use a signed URL instead).

https://www.rfc-editor.org/rfc/rfc8058#section-3.1
https://www.rfc-editor.org/rfc/rfc8058#section-8.1
2022-09-29 04:36:11 -05:00
nonamethanks
5051c6649d Tumblr: parse new dashboard links 2022-09-28 17:00:08 +02:00
evazion
4c03ea5be3 Fix #5132: Modqueue displays active posts when excluding any search term
Fix a bug where searching for a negated tag inside the modqueue would show
active posts.

The bug was that in a search like this:

    Post.in_modqueue.user_tag_match("-solo")

The `in_modqueue` condition would get sucked inside the tag search and negated
when we tried to apply the negation operator to the "solo" tag. So effectively
the `in_modqueue` condition would get negated and we would end up searching for
everything not in the modqueue.
2022-09-28 00:29:50 -05:00