Commit Graph

10769 Commits

Author SHA1 Message Date
evazion
dccc2edb75 tests: fix broken tests.
* Fix a Twitter test broken by a privated tweet.
* Fix an IP geolocation test broken by the ipregistry.co API returning new data.
2021-11-02 04:42:07 -05:00
evazion
9a9649bee8 BURs: change default order on index page to newest BURs first.
Change the /bulk_update_requests page to show the newest BURs first
instead of pending BURs first. This is more consistent with index pages
on the rest of the site, which normally default to newest first. Fixes
an issue where failed BURs would be shown first forever unless they were
manually approved or rejected.
2021-11-02 04:16:33 -05:00
evazion
148752d3c4 PostQueryBuilder: remove useless code.
The workaround for `unaliased:fav:1` is no longer needed since favorites
are no longer included in the post's tag_index.
2021-11-02 04:07:21 -05:00
evazion
4b94cac757 Merge pull request #4911 from nonamethanks/feat-plurk
Add Plurk support
2021-11-02 04:07:07 -05:00
evazion
d8a55bd62b comments: fix video duration not being shown on thumbnails
Fix the video duration not being shown on thumbnails on the
https://danbooru.donmai.us/comments page.

BUG: this introduces duplicate HTML ids on the comments page. Post
thumbnails and post comment containers both have the same html ID.
2021-11-02 03:46:48 -05:00
evazion
a5ed8c72c9 search: fix parsing of invalid metatag values.
* Change `age:` metatag to require time units. This means e.g.
  `age:<600` no longer works; instead you have to say `age:<600sec`.

* Allow time units in the `age:` metatag to be abbreviated as long as
  they're unambiguous. This means `age:<60sec`, `age:<5min`, and
  `age:<5mon` now work, in addition to `age:<60s` and `age:<60seconds`.

* Allow the `ratio:` metatag to be written like `ratio:16/9` in addition
  to `ratio:16:9`.

* Fix invalid date searches like `date:foo` or `date:05-15-2021`
  to return nothing instead of raising an "undefined method
  'beginning_of_day' for nil" exception. (`date:05-15-2021` is invalid
  because it's parsed as DD-MM-YYYY).

* Fix invalid searches like `score:foo`, `ratio:foo`, and `mpixels:foo`
  to return nothing instead of being treated like `score:0`, `ratio:0`,
  `mpixels:0`.

* Fix `age:<60m` to return nothing instead of silently being treated
  like `age:<60seconds`.

* Fix `age:foo` to return nothing instead of silently being treated like
  `age:0d` (return all uploads from today).

Fixes #4389.
2021-11-02 01:54:05 -05:00
nonamethanks
060223c9e2 Add Plurk support 2021-11-01 16:21:27 +01:00
evazion
788dcbd87b Temp disable dumping favorites table to BigQuery.
The favorites table is too big and dumping it tends to time out. Then
the job keeps retrying even though it always fails, then multiple
instances of the job build up in the job queue because the old jobs
never finish.
2021-11-01 05:15:31 -05:00
evazion
5177a28f2c Merge pull request #4910 from nonamethanks/feat-foundation
Add Foundation support
2021-11-01 05:07:44 -05:00
nonamethanks
043f2fb124 Add Foundation support 2021-11-01 01:39:56 +01:00
evazion
9ff4d94382 Merge pull request #4909 from nonamethanks/add-lofter-theme
Lofter: Add support for additional theme
2021-10-31 05:13:04 -05:00
nonamethanks
5946544f71 Lofter: Add support for additional theme 2021-10-30 17:22:45 +02:00
evazion
f593828bb9 storage manager: refactor base_dir option.
Fix it so the `base_dir` option is only required by subclasses that
actually use it. The StorageManager::Mirror class doesn't use it.
2021-10-29 07:14:21 -05:00
evazion
4095d14f2a media assets: fix tagged filenames option.
Fix the `enable_seo_post_urls` config option not being respected. This
option controls whether filenames in image URLs contain the tags. This
option requires URLs rewrites in Nginx to work so it's disabled by
default.
2021-10-29 07:14:21 -05:00
evazion
1614b301e3 storage managers: add mirror storage manager.
Add a storage manager that allows mirroring files to multiple storage
backends.
2021-10-29 07:14:21 -05:00
evazion
e1ecc37c71 Merge pull request #4907 from nottalulah/patch-2
remove unused confirm_ban route
2021-10-28 19:38:47 -05:00
Lily
66551e7ff2 remove unused confirm_ban route
The associated action was removed in e29e9eda49 so this route causes an error in development and a blank page in production.
2021-10-28 21:05:55 -03:00
evazion
26fe3e26e0 tests: silence output from pg_amcheck.
Fix pg_amcheck flooding the test suite output in Github.
2021-10-27 04:59:21 -05:00
evazion
acbbb7a520 news updates: remove the timestamp; change close icon.
Remove the timestamp because it takes up too much space and isn't always needed.
2021-10-27 04:25:22 -05:00
evazion
3b44e08783 docker: update ffmpeg and libvips. 2021-10-27 04:21:01 -05:00
evazion
94ad853a5a docker: add rclone to base Docker image.
Add Rclone to the Docker image so the Rclone storage manager is usable.
2021-10-27 03:50:34 -05:00
evazion
a607cb1cb1 posts: fix IP leak in /posts.json includes.
This leaks the full post object in the error message, which includes the
uploader's IP:

* https://danbooru.donmai.us/posts/4871548.json?only=updater
2021-10-27 02:36:24 -05:00
evazion
e697d1886d Fix #4899: Alias fails when implication already exists. 2021-10-27 01:18:54 -05:00
evazion
eddff747d6 Fix certain IPs not being recognized as proxies.
Fix certain IPs (namely Digital Ocean IPs) no longer being recognized as
proxy IPs by the Ipregistry.co API. Caused by some sudden change in the
API.
2021-10-27 00:05:44 -05:00
evazion
082544ab03 StorageManager: remove Post-specific code.
Refactor StorageManager to remove all image URL generation code. Instead
the image URL generation code lives in MediaAsset.

Now StorageManager is only concerned with how to read and write files to
remote storage backends like S3 or SFTP, not with how image URLs should
be generated. This way the file storage code isn't tightly coupled to
posts, so it can be used to store any kind of file, not just images
belonging to posts.
2021-10-27 00:05:30 -05:00
evazion
afe5095ee6 posts: mark media asset as expunged when post is expunged.
Fix it so that when a post is expunged, the media asset is also marked
as expunged. This way the files will be deleted, but the media asset
will still remain as a record of what was expunged. The media asset will
have the md5, width, height, file ext, and file size of the deleted file.
2021-10-26 02:53:32 -05:00
evazion
84212acfae Merge pull request #4905 from nottalulah/remove-locks-from-autocomplete
remove references to locks
2021-10-25 21:18:36 -05:00
evazion
af5cd83cd7 posts: adjust css of thumbnail video length indicator.
Switch the font to 11px bold Arial. This is more compact and more
readable than 9px Tahoma. Also add a slight border radius and margins
around the indicator to make it stand out from the edge of the image.
2021-10-25 21:17:39 -05:00
evazion
094ed4c11d css: standardize font sizes.
Fix various elements to use standard font sizes instead of ad-hoc sizes.

Noticeable changes:

* Tags in autocomplete are slightly smaller.
* The favorite heart icon on posts is slightly smaller.
* Pool titles on thumbnails in the pool gallery page are slightly bigger.
* The page footer is slightly smaller.
* Timestamps on comments and forum posts are very slightly smaller.
* "Pending"/"approved"/"rejected" labels on forum posts are very slightly smaller.
2021-10-25 05:59:40 -05:00
evazion
7034872132 css: use rem instead of em units instead for font sizes.
Use rem units for font sizes so that font sizes are relative to the root
<html> element, not the parent element.

Fixes an issue where the video duration indicator would be too small on
parent/child thumbnails in post show pages. This was because of nesting
issues with em units. Em units are relative to their parent element, so
if you had a parent element with a font size of 0.8em, and a child
element with a font size of 0.8em, then the final computed font size
would be 0.8*0.8 = 0.64em.
2021-10-25 05:16:59 -05:00
evazion
f1b5c34b4d posts: show length of videos and animations in thumbnails.
Show the length of videos and animated posts in the thumbnail. The
length is shown the top left corner in MM:SS format. This replaces the
play button icon.

Show a speaker icon instead of a music note icon for posts with sound.

Doing this requires doing `.includes(:media_asset)` in a bunch of
places to avoid N+1 queries when we access the post's duration.
2021-10-25 02:56:55 -05:00
evazion
be505920d1 media assets: add script to fix assets with deleted files.
Mark assets that have missing files as expunged. This happened with
uploads that were abandoned and had their files deleted, but that didn't
destroy their media asset record.

Fixes an issue where uploads could have missing files because someone
resumed an abandoned upload that had its files deleted.
2021-10-24 23:00:00 -05:00
evazion
6d2ce5c8c1 saved searches: lower job priority.
Lower the priority of the populate saved search job. This is so that
large numbers of saved searches don't overwhelm the job queue and
prevent higher priority jobs from running.
2021-10-24 22:56:59 -05:00
evazion
a9088d8a87 search: fix flag_count:N metatag being broken. 2021-10-24 17:02:38 -05:00
Lily
647848b499 remove references to locks 2021-10-24 15:16:48 -03:00
evazion
8d5e0a5b58 replacements: don't delete replaced files.
Don't delete replaced files after 30 days. There are only about 30k
replacements in total, so the cost of keeping replaced files is
negligible. It was also wrong because the media asset wasn't destroyed
too, so there were active media assets with missing files.
2021-10-24 04:35:13 -05:00
evazion
d258790199 uploads: don't delete files of abandoned uploads.
Just leave them. They don't take up that much space and they may be used
in the future if someone else tries to upload the same file.
2021-10-24 04:35:13 -05:00
evazion
f5e7d50dbb media assets: don't destroy ugoira data on destroy.
Don't destroy Pixiv Ugoira frame data when the media asset is destroyed.
This is wrong because when uploads were pruned, it could delete the
frame data of an active post.
2021-10-24 04:35:13 -05:00
evazion
5c7a0f225c media assets: prevent duplicate media assets.
Add a md5 uniqueness constraint on media assets to prevent duplicate
assets from being created. This way we can guarantee that there is one
active media asset per uploaded file.

Also make it so that if two people are uploading the same file at the
same time, the file is processed only once.
2021-10-24 04:35:06 -05:00
evazion
a58aa8efa7 Update Ruby gems and Yarn packages. 2021-10-21 03:27:59 -05:00
evazion
f78378cc69 recommendations: reduce cache lifetime to 5 minutes.
These calls aren't actually slow and don't need to be cached for long
periods of time.
2021-10-21 00:46:47 -05:00
evazion
b2d9ff4780 Merge pull request #4903 from Official-NotaNorthKorean/patch-1
Update INSTALL.debian
2021-10-19 16:57:15 -05:00
NotaNorthKorean
aa2db59a0b Update INSTALL.debian
Removed instructions for files which no longer exist
2021-10-19 07:45:57 -07:00
evazion
0221ecdf29 uploads: remove useless code. 2021-10-18 06:25:02 -05:00
evazion
748fdf33d4 uploads: don't autotag sound on videos.
Don't automatically add the sound tag to videos. This was incorrect
nearly 20% of the time because of silent audio tracks. This error rate
is too high.

https://danbooru.donmai.us/posts?tags=exif:Track2:AudioChannels+-sound
https://danbooru.donmai.us/posts?tags=exif:Track1:AudioChannels+-sound
2021-10-18 06:16:47 -05:00
evazion
bc506ed1b8 uploads: refactor to simplify ugoira-handling and replacements:
* Make it so replacing a post doesn't generate a dummy upload as a side effect.
* Make it so you can't replace a post with itself (the post should be regenerated instead).
* Refactor uploads and replacements to save the ugoira frame data when
  the MediaAsset is created, not when the post is created. This way it's
  possible to view the ugoira before the post is created.
* Make `download_file!` in the Pixiv source strategy return a MediaFile
  with the ugoira frame data already attached to it, instead of returning it
  in the `data` field then passing it around separately in the `context`
  field of the upload.
2021-10-18 05:18:46 -05:00
evazion
85c3b4f2d1 ugoiras: add md5 column to pixiv_ugoira_frame_data.
This is necessary so we can associate ugoira frame data with the media
asset instead of with the post.
2021-10-18 00:34:24 -05:00
evazion
1d034a3223 media assets: move more file-handling logic into MediaAsset.
Move more of the file-handling logic from UploadService and
StorageManager into MediaAsset. This is part of refactoring posts and
uploads to allow multiple images per post.
2021-10-18 00:10:29 -05:00
evazion
8b3ab04724 media file: fix calculation of video/animation duration.
Fix how the duration of videos and animated GIFs / PNGs is calculated.
If we can't determine the duration from the file metadata, then play the
entire video or animation back using FFmpeg and scrape the duration and
frame count.

This is necessary for things like WebM files where the duration metadata
is optional, or animated GIFs and PNGs that don't have a duration field
in the metadata, only a frame count and a sequence of frame delays.
2021-10-17 20:15:51 -05:00
evazion
d8de58d991 Fix bug in 079_fix_duration.rb
`assets` was unused.
2021-10-17 18:31:54 -05:00