uploads: rework upload process.
Rework the upload process so that files are saved to Danbooru first before the user starts tagging the upload. The main user-visible change is that you have to select the file first before you can start tagging it. Saving the file first lets us fix a number of problems: * We can check for dupes before the user tags the upload. * We can perform dupe checks and show preview images for users not using the bookmarklet. * We can show preview images without having to proxy images through Danbooru. * We can show previews of videos and ugoira files. * We can reliably show the filesize and resolution of the image. * We can let the user save files to upload later. * We can get rid of a lot of spaghetti code related to preprocessing uploads. This was the cause of most weird "md5 confirmation doesn't match md5" errors. (Not all of these are implemented yet.) Internally, uploading is now a two-step process: first we create an upload object, then we create a post from the upload. This is how it works: * The user goes to /uploads/new and chooses a file or pastes an URL into the file upload component. * The file upload component calls `POST /uploads` to create an upload. * `POST /uploads` immediately returns a new upload object in the `pending` state. * Danbooru starts processing the upload in a background job (downloading, resizing, and transferring the image to the image servers). * The file upload component polls `/uploads/$id.json`, checking the upload `status` until it returns `completed` or `error`. * When the upload status is `completed`, the user is redirected to /uploads/$id. * On the /uploads/$id page, the user can tag the upload and submit it. * The upload form calls `POST /posts` to create a new post from the upload. * The user is redirected to the new post. This is the data model: * An upload represents a set of files uploaded to Danbooru by a user. Uploaded files don't have to belong to a post. An upload has an uploader, a status (pending, processing, completed, or error), a source (unless uploading from a file), and a list of media assets (image or video files). * There is a has-and-belongs-to-many relationship between uploads and media assets. An upload can have many media assets, and a media asset can belong to multiple uploads. Uploads are joined to media assets through a upload_media_assets table. An upload could potentially have multiple media assets if it's a Pixiv or Twitter gallery. This is not yet implemented (at the moment all uploads have one media asset). A media asset can belong to multiple uploads if multiple people try to upload the same file, or if the same user tries to upload the same file more than once. New features: * On the upload page, you can press Ctrl+V to paste an URL and immediately upload it. * You can save files for upload later. Your saved files are at /uploads. Fixes: * Improved error messages when uploading invalid files, bad URLs, and when forgetting the rating.
This commit is contained in:
evazion
@@ -4,6 +4,10 @@ class MediaAsset < ApplicationRecord
|
||||
class Error < StandardError; end
|
||||
|
||||
VARIANTS = %i[preview 180x180 360x360 720x720 sample original]
|
||||
MAX_VIDEO_DURATION = 140 # 2:20
|
||||
MAX_IMAGE_RESOLUTION = Danbooru.config.max_image_resolution
|
||||
MAX_IMAGE_WIDTH = Danbooru.config.max_image_width
|
||||
MAX_IMAGE_HEIGHT = Danbooru.config.max_image_height
|
||||
ENABLE_SEO_POST_URLS = Danbooru.config.enable_seo_post_urls
|
||||
LARGE_IMAGE_WIDTH = Danbooru.config.large_image_width
|
||||
STORAGE_SERVICE = Danbooru.config.storage_manager
|
||||
@@ -30,6 +34,9 @@ class MediaAsset < ApplicationRecord
|
||||
}
|
||||
|
||||
validates :md5, uniqueness: { conditions: -> { where(status: [:processing, :active]) } }
|
||||
validates :file_ext, inclusion: { in: %w[jpg png gif mp4 webm swf zip], message: "Not an image or video" }
|
||||
validates :duration, numericality: { less_than_or_equal_to: MAX_VIDEO_DURATION, message: "must be less than #{MAX_VIDEO_DURATION} seconds", allow_nil: true }, on: :create # XXX should allow admins to bypass
|
||||
validate :validate_resolution, on: :create
|
||||
|
||||
class Variant
|
||||
extend Memoist
|
||||
@@ -194,6 +201,8 @@ class MediaAsset < ApplicationRecord
|
||||
# This can't be called inside a transaction because the transaction will
|
||||
# fail if there's a RecordNotUnique error when the asset already exists.
|
||||
def upload!(media_file)
|
||||
raise Error, "File is corrupt" if media_file.is_corrupt?
|
||||
|
||||
media_asset = create!(file: media_file, status: :processing)
|
||||
media_asset.distribute_files!(media_file)
|
||||
media_asset.update!(status: :active)
|
||||
@@ -318,4 +327,18 @@ class MediaAsset < ApplicationRecord
|
||||
is_animated? && file_ext == "png"
|
||||
end
|
||||
end
|
||||
|
||||
concerning :ValidationMethods do
|
||||
def validate_resolution
|
||||
resolution = image_width * image_height
|
||||
|
||||
if resolution > MAX_IMAGE_RESOLUTION
|
||||
errors.add(:base, "Image resolution is too large (resolution: #{(resolution / 1_000_000.0).round(1)} megapixels (#{image_width}x#{image_height}); max: #{MAX_IMAGE_RESOLUTION / 1_000_000} megapixels)")
|
||||
elsif image_width > MAX_IMAGE_WIDTH
|
||||
errors.add(:image_width, "is too large (width: #{image_width}; max width: #{MAX_IMAGE_WIDTH})")
|
||||
elsif image_height > MAX_IMAGE_HEIGHT
|
||||
errors.add(:image_height, "is too large (height: #{image_height}; max height: #{MAX_IMAGE_HEIGHT})")
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
@@ -18,8 +18,9 @@ class Post < ApplicationRecord
|
||||
before_validation :parse_pixiv_id
|
||||
before_validation :blank_out_nonexistent_parents
|
||||
before_validation :remove_parent_loops
|
||||
validates :md5, uniqueness: { message: ->(post, _data) { "duplicate: #{Post.find_by_md5(post.md5).id}" }}, on: :create
|
||||
validates :rating, inclusion: { in: %w[s q e], message: "rating must be s, q, or e" }
|
||||
validates :md5, uniqueness: { message: ->(post, _data) { "Duplicate of post ##{Post.find_by_md5(post.md5).id}" }}, on: :create
|
||||
validates :rating, presence: { message: "not selected" }
|
||||
validates :rating, inclusion: { in: %w[s q e], message: "must be S, Q, or E" }, if: -> { rating.present? }
|
||||
validates :source, length: { maximum: 1200 }
|
||||
validate :added_tags_are_valid
|
||||
validate :removed_tags_are_valid
|
||||
@@ -34,6 +35,7 @@ class Post < ApplicationRecord
|
||||
after_save :create_version
|
||||
after_save :update_parent_on_save
|
||||
after_save :apply_post_metatags
|
||||
after_create_commit :update_iqdb
|
||||
|
||||
belongs_to :approver, class_name: "User", optional: true
|
||||
belongs_to :uploader, :class_name => "User", :counter_cache => "post_upload_count"
|
||||
@@ -73,6 +75,36 @@ class Post < ApplicationRecord
|
||||
has_many :versions, -> { Rails.env.test? ? order("post_versions.updated_at ASC, post_versions.id ASC") : order("post_versions.updated_at ASC") }, class_name: "PostVersion", dependent: :destroy
|
||||
end
|
||||
|
||||
def self.new_from_upload(params)
|
||||
upload_media_asset = UploadMediaAsset.find(params[:upload_media_asset_id])
|
||||
media_asset = upload_media_asset.media_asset
|
||||
upload = upload_media_asset.upload
|
||||
|
||||
# XXX depends on CurrentUser
|
||||
commentary = ArtistCommentary.new(
|
||||
original_title: params[:artist_commentary_title],
|
||||
original_description: params[:artist_commentary_desc],
|
||||
translated_title: params[:translated_commentary_title],
|
||||
translated_description: params[:translated_commentary_desc],
|
||||
)
|
||||
|
||||
post = Post.new(
|
||||
uploader: upload.uploader,
|
||||
uploader_ip_addr: upload.uploader_ip_addr,
|
||||
md5: media_asset.md5,
|
||||
file_ext: media_asset.file_ext,
|
||||
file_size: media_asset.file_size,
|
||||
image_width: media_asset.image_width,
|
||||
image_height: media_asset.image_height,
|
||||
source: Sources::Strategies.find(upload.source, upload.referer_url).canonical_url || upload.source,
|
||||
tag_string: params[:tag_string],
|
||||
rating: params[:rating],
|
||||
parent_id: params[:parent_id],
|
||||
is_pending: !upload.uploader.can_upload_free? || params[:is_pending].to_s.truthy?,
|
||||
artist_commentary: (commentary if commentary.any_field_present?),
|
||||
)
|
||||
end
|
||||
|
||||
module FileMethods
|
||||
extend ActiveSupport::Concern
|
||||
|
||||
|
||||
@@ -1,100 +1,28 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
class Upload < ApplicationRecord
|
||||
class Error < StandardError; end
|
||||
|
||||
MAX_VIDEO_DURATION = 140
|
||||
|
||||
class FileValidator < ActiveModel::Validator
|
||||
def validate(record)
|
||||
validate_file_ext(record)
|
||||
validate_integrity(record)
|
||||
validate_md5_uniqueness(record)
|
||||
validate_video_duration(record)
|
||||
validate_resolution(record)
|
||||
end
|
||||
attr_accessor :file
|
||||
|
||||
def validate_file_ext(record)
|
||||
if record.file_ext.in?(["bin", "swf"])
|
||||
record.errors.add(:file_ext, "is invalid (only JPEG, PNG, GIF, MP4, and WebM files are allowed")
|
||||
end
|
||||
end
|
||||
belongs_to :uploader, class_name: "User"
|
||||
has_many :upload_media_assets, dependent: :destroy
|
||||
has_many :media_assets, through: :upload_media_assets
|
||||
|
||||
def validate_integrity(record)
|
||||
if record.file.is_corrupt?
|
||||
record.errors.add(:file, "is corrupted")
|
||||
end
|
||||
end
|
||||
validates :source, format: { with: %r{\Ahttps?://}i, message: "is not a valid URL" }, if: -> { source.present? }
|
||||
validates :referer_url, format: { with: %r{\Ahttps?://}i, message: "is not a valid URL" }, if: -> { referer_url.present? }
|
||||
|
||||
def validate_md5_uniqueness(record)
|
||||
if record.md5.nil?
|
||||
return
|
||||
end
|
||||
|
||||
md5_post = Post.find_by_md5(record.md5)
|
||||
|
||||
if md5_post.nil?
|
||||
return
|
||||
end
|
||||
|
||||
if record.replaced_post && record.replaced_post == md5_post
|
||||
return
|
||||
end
|
||||
|
||||
record.errors.add(:md5, "duplicate: #{md5_post.id}")
|
||||
end
|
||||
|
||||
def validate_resolution(record)
|
||||
resolution = record.image_width.to_i * record.image_height.to_i
|
||||
|
||||
if resolution > Danbooru.config.max_image_resolution
|
||||
record.errors.add(:base, "image resolution is too large (resolution: #{(resolution / 1_000_000.0).round(1)} megapixels (#{record.image_width}x#{record.image_height}); max: #{Danbooru.config.max_image_resolution / 1_000_000} megapixels)")
|
||||
elsif record.image_width > Danbooru.config.max_image_width
|
||||
record.errors.add(:image_width, "is too large (width: #{record.image_width}; max width: #{Danbooru.config.max_image_width})")
|
||||
elsif record.image_height > Danbooru.config.max_image_height
|
||||
record.errors.add(:image_height, "is too large (height: #{record.image_height}; max height: #{Danbooru.config.max_image_height})")
|
||||
end
|
||||
end
|
||||
|
||||
def validate_video_duration(record)
|
||||
if !record.uploader.is_admin? && record.file.is_video? && record.file.duration.to_i > MAX_VIDEO_DURATION
|
||||
record.errors.add(:base, "video must not be longer than #{MAX_VIDEO_DURATION.seconds.inspect}")
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
attr_accessor :as_pending, :replaced_post, :file
|
||||
|
||||
belongs_to :uploader, :class_name => "User"
|
||||
belongs_to :post, optional: true
|
||||
has_one :media_asset, foreign_key: :md5, primary_key: :md5
|
||||
|
||||
before_validation :initialize_attributes, on: :create
|
||||
before_validation :assign_rating_from_tags
|
||||
# validates :source, format: { with: /\Ahttps?/ }, if: ->(record) {record.file.blank?}, on: :create
|
||||
validates :rating, inclusion: { in: %w[q e s] }, allow_nil: true
|
||||
validates :md5, confirmation: true, if: ->(rec) { rec.md5_confirmation.present? }
|
||||
validates_with FileValidator, on: :file
|
||||
serialize :context, JSON
|
||||
after_create :async_process_upload!
|
||||
|
||||
scope :pending, -> { where(status: "pending") }
|
||||
scope :preprocessed, -> { where(status: "preprocessed") }
|
||||
scope :completed, -> { where(status: "completed") }
|
||||
scope :uploaded_by, ->(user_id) { where(uploader_id: user_id) }
|
||||
|
||||
def initialize_attributes
|
||||
self.uploader_id = CurrentUser.id
|
||||
self.uploader_ip_addr = CurrentUser.ip_addr
|
||||
self.server = Socket.gethostname
|
||||
end
|
||||
|
||||
def self.visible(user)
|
||||
if user.is_admin?
|
||||
all
|
||||
elsif user.is_anonymous?
|
||||
completed
|
||||
else
|
||||
completed.or(where(uploader: user))
|
||||
where(uploader: user)
|
||||
end
|
||||
end
|
||||
|
||||
@@ -111,98 +39,44 @@ class Upload < ApplicationRecord
|
||||
status == "completed"
|
||||
end
|
||||
|
||||
def is_preprocessed?
|
||||
status == "preprocessed"
|
||||
end
|
||||
|
||||
def is_preprocessing?
|
||||
status == "preprocessing"
|
||||
end
|
||||
|
||||
def is_duplicate?
|
||||
status.match?(/duplicate: \d+/)
|
||||
end
|
||||
|
||||
def is_errored?
|
||||
status.match?(/error:/)
|
||||
end
|
||||
|
||||
def sanitized_status
|
||||
if is_errored?
|
||||
status.sub(/DETAIL:.+/m, "...")
|
||||
else
|
||||
status
|
||||
end
|
||||
end
|
||||
|
||||
def duplicate_post_id
|
||||
@duplicate_post_id ||= status[/duplicate: (\d+)/, 1]
|
||||
end
|
||||
end
|
||||
|
||||
concerning :SourceMethods do
|
||||
def source=(source)
|
||||
source = source.unicode_normalize(:nfc)
|
||||
|
||||
# percent encode unicode characters in urls
|
||||
if source =~ %r{\Ahttps?://}i
|
||||
source = Addressable::URI.normalized_encode(source) rescue source
|
||||
end
|
||||
|
||||
super(source)
|
||||
end
|
||||
|
||||
def source_url
|
||||
return nil unless source =~ %r{\Ahttps?://}i
|
||||
Addressable::URI.heuristic_parse(source) rescue nil
|
||||
end
|
||||
end
|
||||
|
||||
def self.search(params)
|
||||
q = search_attributes(params, :id, :created_at, :updated_at, :source, :rating, :parent_id, :server, :md5, :server, :file_ext, :file_size, :image_width, :image_height, :referer_url, :uploader, :post)
|
||||
|
||||
if params[:source_matches].present?
|
||||
q = q.where_like(:source, params[:source_matches])
|
||||
end
|
||||
|
||||
if params[:has_post].to_s.truthy?
|
||||
q = q.where.not(post_id: nil)
|
||||
elsif params[:has_post].to_s.falsy?
|
||||
q = q.where(post_id: nil)
|
||||
end
|
||||
|
||||
if params[:status].present?
|
||||
q = q.where_like(:status, params[:status])
|
||||
end
|
||||
|
||||
if params[:backtrace].present?
|
||||
q = q.where_like(:backtrace, params[:backtrace])
|
||||
end
|
||||
|
||||
if params[:tag_string].present?
|
||||
q = q.where_like(:tag_string, params[:tag_string])
|
||||
end
|
||||
|
||||
q = search_attributes(params, :id, :created_at, :updated_at, :source, :referer_url, :uploader, :status, :backtrace, :upload_media_assets, :media_assets)
|
||||
q.apply_default_order(params)
|
||||
end
|
||||
|
||||
def assign_rating_from_tags
|
||||
rating = PostQueryBuilder.new(tag_string).find_metatag(:rating)
|
||||
|
||||
if rating.present?
|
||||
self.rating = rating.downcase.first
|
||||
def async_process_upload!
|
||||
if file.present?
|
||||
ProcessUploadJob.perform_now(self)
|
||||
else
|
||||
ProcessUploadJob.perform_later(self)
|
||||
end
|
||||
end
|
||||
|
||||
def upload_as_pending?
|
||||
as_pending.to_s.truthy?
|
||||
end
|
||||
def process_upload!
|
||||
update!(status: "processing")
|
||||
|
||||
def has_commentary?
|
||||
artist_commentary_title.present? || artist_commentary_desc.present? || translated_commentary_title.present? || translated_commentary_desc.present?
|
||||
if file.present?
|
||||
media_file = MediaFile.open(file.tempfile)
|
||||
elsif source.present?
|
||||
strategy = Sources::Strategies.find(source, referer_url)
|
||||
media_file = strategy.download_file!(strategy.image_url)
|
||||
else
|
||||
raise "No file or source provided"
|
||||
end
|
||||
|
||||
media_asset = MediaAsset.upload!(media_file)
|
||||
update!(media_assets: [media_asset], status: "completed")
|
||||
rescue Exception => e
|
||||
update!(status: "error: #{e.message}", backtrace: e.backtrace.join("\n"))
|
||||
raise
|
||||
end
|
||||
|
||||
def self.available_includes
|
||||
[:uploader, :post]
|
||||
[:uploader, :upload_media_assets, :media_assets]
|
||||
end
|
||||
end
|
||||
|
||||
Reference in New Issue
Block a user