Commit Graph

50 Commits

Author SHA1 Message Date
evazion
a7dc05ce63 Enable frozen string literals.
Make all string literals immutable by default.
2021-12-14 21:33:27 -06:00
evazion
bc506ed1b8 uploads: refactor to simplify ugoira-handling and replacements:
* Make it so replacing a post doesn't generate a dummy upload as a side effect.
* Make it so you can't replace a post with itself (the post should be regenerated instead).
* Refactor uploads and replacements to save the ugoira frame data when
  the MediaAsset is created, not when the post is created. This way it's
  possible to view the ugoira before the post is created.
* Make `download_file!` in the Pixiv source strategy return a MediaFile
  with the ugoira frame data already attached to it, instead of returning it
  in the `data` field then passing it around separately in the `context`
  field of the upload.
2021-10-18 05:18:46 -05:00
evazion
3d660953d4 Add MediaMetadata model.
Add a model for storing image and video metadata for uploaded files.

Metadata is extracted using ExifTool. You will need to install ExifTool
after this commit. ExifTool 12.22 is the minimum required version
because we use the `--binary` option, which was added in this release.

The MediaMetadata model is separate from the MediaAsset model because
some files contain tons of metadata, and most of it is non-essential.
The MediaAsset model represents an uploaded file and contains essential
metadata, like the file's size and type, while the MediaMetadata model
represents all the other non-essential metadata associated with a file.

Metadata is stored as a JSON column in the database.

ExifTool returns all the file's metadata, not just the EXIF metadata.
EXIF is one of several types of image metadata, hence why we call
it MediaMetadata instead of EXIFMetadata.
2021-09-08 05:00:54 -05:00
evazion
b068c113a8 Add MediaAsset model.
A MediaAsset represents an image or video file uploaded to Danbooru. It
stores the metadata associated with the image or video. This is to work
on decoupling files from posts so that images can be uploaded separately
from posts.
2021-09-02 06:07:52 -05:00
evazion
00ca7526bb docs: add remaining docs for classes in app/logical. 2021-06-24 01:31:41 -05:00
evazion
e5cfb7904c CurrentUser: remove #as method.
Replace with CurrentUser#scoped.
2021-06-22 23:39:30 -05:00
evazion
07e23204b6 rubocop: fix various Rubocop warnings. 2021-06-17 04:17:53 -05:00
evazion
0f36bbf8d3 iqdb: update API client to use new version of IQDB.
Replace the old IQDB API client with a new client for the new forked
version of IQDB at https://github.com/danbooru/iqdb.

Changes:

* The /iqdb_queries endpoint now returns `hash` and `signature` fields.
  The `signature` is the full decoded Haar signature, while the `hash`
  is a encoded version of the signature.
* The /iqdb_queries endpoint no longer returns `width` and `height`
  fields in the response (these were always 128x128).
* We no longer need the IQDBs frontend server, now we talk to the IQDB
  instance directly.
* We no longer send add/remove image commands to IQDB through AWS SQS,
  now we send them to IQDB directly. They are sent in a delayed job so
  that if IQDB is down, uploading images is still possible, the add
  image commands will just get queued up.
* Fix a bug where regenerating an image's thumbnails didn't regenerate
  IQDB, because IQDB silently ignored add image commands when the image
  already existed in the database.
2021-06-16 05:36:24 -05:00
evazion
b551e3634f Fix misc rubocop warnings. 2020-06-16 21:36:15 -05:00
evazion
e477232e02 uploads: factor out image dimension and filetype detection code.
* Add MediaFile abstraction. A MediaFile represents an image or video file.
* Move filetype detection and dimension parsing code from uploads to MediaFile.
2020-05-06 00:33:35 -05:00
evazion
743b6f0854 Fix #4377: Save commentary by default.
Remove the "Include artist commentary" checkbox. Commentary is included
by default unless the commentary fields are blank.
2020-04-04 00:46:36 -05:00
evazion
ab1839c613 uploads: fix exception with preprocessed uploads.
Fix exception when submitting an upload and an in-progress preprocessed
upload already exists. In this case we forgot to pass the upload params
when calling UploadService#delayed_start.
2020-03-31 21:57:34 -05:00
BrokenEagle
3a422b26df Removed unneeded commentary tag checkboxes
As per the discussion on GitHub, these will be handled instead
by the existing mechanisms of related and frequent tags.
2020-01-16 21:40:14 +00:00
BrokenEagle
82b621d87d Added upload commentary enhancements
- Can now translate commentary from the upload page
- Can now add commentary tags with a checkbox
2020-01-14 23:59:27 +00:00
evazion
309821bf73 rubocop: fix various style issues. 2019-12-22 21:23:37 -06:00
evazion
cb45aa6fe6 Remove unused automod code.
The DynamoDB update job was disabled a few months ago. Remove it
completely.
2019-12-03 10:45:18 -06:00
evazion
0a6661d145 uploads: switch to active job.
* Switch upload processing from DelayedJob to ActiveJob.
* Remove remaining references to delayed job from tests.

Closes #4128.
2019-09-23 15:11:18 -05:00
Albert Yi
2961be636a add job hook when post is created 2019-01-30 10:54:27 -08:00
evazion
a20eba9ef7 Fix #3983: Uploads: NoMethodError - undefined method `>=' for nil:NilClass
* Rename Upload#download_for_upload to #get_file_for_upload.

* Fix #get_file_for_upload to raise error if no file or source url was given.

* Fix javascript upload validation to disallow submitting form if file is
  not present and the source is not an url.
2018-11-11 17:12:39 -06:00
evazion
e10c6c6a30 Fix #3982: Uploads: URI::InvalidURIError - URI must be ascii only. 2018-11-11 12:17:58 -06:00
Albert Yi
55cbef52e4 add default source when creating posts from uploads in case canonical url doesn't exist 2018-08-29 14:21:39 -07:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
r888888888
cc7a66438f expose recommended posts to everyone 2018-07-23 15:05:43 -07:00
Albert Yi
ac28c92fbd refactor upload service 2018-07-06 11:52:43 -07:00
Albert Yi
a9896031df more fault tolerance for uploads 2018-07-05 15:59:06 -07:00
Albert Yi
d9e15cbf31 add user scoping for upload start 2018-07-05 10:07:41 -07:00
Albert Yi
6dfb78fca8 split out image cropping to width x height, add StorageManager::Match 2018-07-03 17:59:53 -07:00
Albert Yi
99807580d5 add cropped support for video uploads 2018-07-03 16:37:53 -07:00
Albert Yi
867406f410 handle the case where a file is uploaded that doesn't match whatever was preprocessed 2018-07-02 14:47:23 -07:00
Albert Yi
9a522d9fef for upload service, move md5 validation to upload/post. exclude self when doing a self replacement. 2018-07-02 13:26:47 -07:00
Albert Yi
4836804cae bubble up upload validation errors during replacement
restore missing UploadService::Replacer tests
addresses #3761
2018-06-25 15:04:29 -07:00
Albert Yi
c17e0baad0 prefer file over source when both are provided (#3767) 2018-06-25 14:00:02 -07:00
Albert Yi
64446d49e1 add image cropping support 2018-06-22 14:41:57 -07:00
Albert Yi
d240fdde66 fix Utils.is_downloadable? for null case 2018-06-20 10:51:29 -07:00
Albert Yi
62218fdb51 move uploader upload count increment from upload to post 2018-06-20 10:43:48 -07:00
Albert Yi
3f6e069eed fixes #3762 2018-06-20 10:07:27 -07:00
Albert Yi
51a11561c3 addresses #3760 2018-06-19 17:20:40 -07:00
Albert Yi
97d0dca8a4 increase timeout of deleting preprocessed images 2018-06-19 13:27:58 -07:00
Albert Yi
a560d61b68 handle preprocessed dups 2018-06-19 12:39:03 -07:00
Albert Yi
35bc9a2e32 capture errors thrown on upload preprocessing create 2018-06-19 11:28:13 -07:00
Albert Yi
578e5a6fea fix bug with deleting uploads 2018-06-19 11:09:56 -07:00
Albert Yi
a64a9fa0d4 additional tweaks to upload preprocessor 2018-06-19 10:51:05 -07:00
Albert Yi
88dec0f592 dup record when distributing files 2018-06-18 15:41:44 -07:00
Albert Yi
cd043c4868 fix bug with files geting uploaded to wrong servers 2018-06-18 14:20:33 -07:00
Albert Yi
cc24b0a54f fix bug with copy-pasted urls in uploads/new erroring out 2018-06-18 10:23:01 -07:00
Albert Yi
a6e972a6e6 abort on dup md5 preprocessing 2018-06-14 17:52:41 -07:00
Albert Yi
b0c2ddba8b update tests 2018-06-14 17:52:41 -07:00
Albert Yi
0e6c358701 add drag and drop file uploads w/async processing
[skip ci]
2018-06-14 17:52:41 -07:00
Albert Yi
e808a4f7cb delete unfinished upload files 2018-06-14 17:52:41 -07:00
Albert Yi
fdd7582fb0 add support for upload preprocessing 2018-06-14 17:52:41 -07:00