jaculabilis/danbooru
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

uploads: rework upload process.

Browse Source 
Rework the upload process so that files are saved to Danbooru first
before the user starts tagging the upload.

The main user-visible change is that you have to select the file first
before you can start tagging it. Saving the file first lets us fix a
number of problems:

* We can check for dupes before the user tags the upload.
* We can perform dupe checks and show preview images for users not using the bookmarklet.
* We can show preview images without having to proxy images through Danbooru.
* We can show previews of videos and ugoira files.
* We can reliably show the filesize and resolution of the image.
* We can let the user save files to upload later.
* We can get rid of a lot of spaghetti code related to preprocessing
  uploads. This was the cause of most weird "md5 confirmation doesn't
  match md5" errors.

(Not all of these are implemented yet.)

Internally, uploading is now a two-step process: first we create an upload
object, then we create a post from the upload. This is how it works:

* The user goes to /uploads/new and chooses a file or pastes an URL into
  the file upload component.
* The file upload component calls `POST /uploads` to create an upload.
* `POST /uploads` immediately returns a new upload object in the `pending` state.
* Danbooru starts processing the upload in a background job (downloading,
  resizing, and transferring the image to the image servers).
* The file upload component polls `/uploads/$id.json`, checking the
  upload `status` until it returns `completed` or `error`.
* When the upload status is `completed`, the user is redirected to /uploads/$id.
* On the /uploads/$id page, the user can tag the upload and submit it.
* The upload form calls `POST /posts` to create a new post from the upload.
* The user is redirected to the new post.

This is the data model:

* An upload represents a set of files uploaded to Danbooru by a user.
  Uploaded files don't have to belong to a post. An upload has an
  uploader, a status (pending, processing, completed, or error), a
  source (unless uploading from a file), and a list of media assets
  (image or video files).

* There is a has-and-belongs-to-many relationship between uploads and
  media assets. An upload can have many media assets, and a media asset
  can belong to multiple uploads. Uploads are joined to media assets
  through a upload_media_assets table.

  An upload could potentially have multiple media assets if it's a Pixiv
  or Twitter gallery. This is not yet implemented (at the moment all
  uploads have one media asset).

  A media asset can belong to multiple uploads if multiple people try
  to upload the same file, or if the same user tries to upload the same
  file more than once.

New features:

* On the upload page, you can press Ctrl+V to paste an URL and immediately upload it.
* You can save files for upload later. Your saved files are at /uploads.

Fixes:

* Improved error messages when uploading invalid files, bad URLs, and
  when forgetting the rating.
This commit is contained in:
evazion
2022-01-26 00:27:47 -06:00
parent f11c46b4f8
commit abdab7a0a8
46 changed files with 621 additions and 1016 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
app/logical/sources/strategies/base.rb
View File

@@ -214,15 +214,6 @@ module Sources
{}
end
# Returns the size of the image resource without actually downloading the file.
def remote_size
response = http_downloader.head(image_url)
return nil unless response.status == 200 && response.content_length.present?
response.content_length.to_i
end
memoize :remote_size
# Download the file at the given url, or at the main image url by default.
def download_file!(download_url = image_url)
raise DownloadError, "Download failed: couldn't find download url for #{url}" if download_url.blank?

106
app/logical/upload_service.rb
View File

@@ -1,106 +0,0 @@
# frozen_string_literal: true
# A service object for uploading an image.
class UploadService
attr_reader :params, :post, :upload
def initialize(params)
@params = params
end
def delayed_start(uploader)
CurrentUser.scoped(uploader) do
start!
end
rescue ActiveRecord::RecordNotUnique
end
def start!
preprocessor = Preprocessor.new(params)
if preprocessor.in_progress?
UploadServiceDelayedStartJob.set(wait: 5.seconds).perform_later(params, CurrentUser.user)
return preprocessor.predecessor
end
if preprocessor.completed?
@upload = preprocessor.finish!
begin
create_post_from_upload(@upload)
rescue Exception => e
@upload.update(status: "error: #{e.class} - #{e.message}", backtrace: e.backtrace.join("\n"))
end
return @upload
end
params[:rating] ||= "q"
params[:tag_string] ||= "tagme"
@upload = Upload.create!(params)
begin
if @upload.invalid?
return @upload
end
@upload.update(status: "processing")
@upload.file = Utils.get_file_for_upload(@upload.source_url, @upload.referer_url, @upload.file&.tempfile)
Utils.process_file(upload, @upload.file)
@upload.save!
@post = create_post_from_upload(@upload)
@upload
rescue Exception => e
@upload.update(status: "error: #{e.class} - #{e.message}", backtrace: e.backtrace.join("\n"))
@upload
end
end
def warnings
return [] if @post.nil?
@post.warnings.full_messages
end
def create_post_from_upload(upload)
@post = convert_to_post(upload)
@post.save!
if upload.has_commentary?
@post.create_artist_commentary(
:original_title => upload.artist_commentary_title,
:original_description => upload.artist_commentary_desc,
:translated_title => upload.translated_commentary_title,
:translated_description => upload.translated_commentary_desc
)
end
@post.update_iqdb
upload.update(status: "completed", post_id: @post.id)
@post
end
def convert_to_post(upload)
Post.new.tap do |p|
p.tag_string = upload.tag_string
p.md5 = upload.md5
p.file_ext = upload.file_ext
p.image_width = upload.image_width
p.image_height = upload.image_height
p.rating = upload.rating
if upload.source.present?
p.source = Sources::Strategies.find(upload.source, upload.referer_url).canonical_url || upload.source
end
p.file_size = upload.file_size
p.uploader_id = upload.uploader_id
p.uploader_ip_addr = upload.uploader_ip_addr
p.parent_id = upload.parent_id
if !upload.uploader.can_upload_free? || upload.upload_as_pending?
p.is_pending = true
end
end
end
end

26
app/logical/upload_service/controller_helper.rb
View File

@@ -1,26 +0,0 @@
# frozen_string_literal: true
class UploadService
module ControllerHelper
def self.prepare(url: nil, file: nil, ref: nil)
upload = Upload.new
if Utils.is_downloadable?(url) && file.nil?
# this gets called from UploadsController#new so we need to preprocess async
UploadPreprocessorDelayedStartJob.perform_later(url, ref, CurrentUser.user)
strategy = Sources::Strategies.find(url, ref)
remote_size = strategy.remote_size
return [upload, remote_size]
end
if file
# this gets called via XHR so we can process sync
Preprocessor.new(file: file).delayed_start(CurrentUser.user)
end
[upload]
end
end
end

95
app/logical/upload_service/preprocessor.rb
View File

@@ -1,95 +0,0 @@
# frozen_string_literal: true
class UploadService
class Preprocessor
extend Memoist
attr_reader :params
def initialize(params)
@params = params
end
def source
params[:source]
end
def md5
params[:md5_confirmation]
end
def referer_url
params[:referer_url]
end
def in_progress?
if md5.present?
Upload.exists?(status: "preprocessing", md5: md5)
elsif Utils.is_downloadable?(source)
Upload.exists?(status: "preprocessing", source: source)
else
false
end
end
def predecessor
if md5.present?
Upload.where(status: ["preprocessed", "preprocessing"], md5: md5).first
elsif Utils.is_downloadable?(source)
Upload.where(status: ["preprocessed", "preprocessing"], source: source).first
end
end
def completed?
predecessor.present?
end
def delayed_start(uploader)
CurrentUser.scoped(uploader) do
start!
end
rescue ActiveRecord::RecordNotUnique
end
def start!
params[:rating] ||= "q"
params[:tag_string] ||= "tagme"
upload = Upload.create!(params)
begin
upload.update(status: "preprocessing")
file = Utils.get_file_for_upload(upload.source_url, upload.referer_url, params[:file]&.tempfile)
Utils.process_file(upload, file)
upload.rating = params[:rating]
upload.tag_string = params[:tag_string]
upload.status = "preprocessed"
upload.save!
rescue Exception => e
upload.update(file_ext: nil, status: "error: #{e.class} - #{e.message}", backtrace: e.backtrace.join("\n"))
end
upload
end
def finish!(upload = nil)
pred = upload || predecessor
# regardless of who initialized the upload, credit should
# goto whoever submitted the form
pred.initialize_attributes
pred.attributes = params
# if a file was uploaded after the preprocessing occurred,
# then process the file and overwrite whatever the preprocessor
# did
Utils.process_file(pred, pred.file.tempfile) if pred.file.present?
pred.status = "completed"
pred.save
pred
end
end
end

12
app/logical/upload_service/replacer.rb
View File

@@ -26,7 +26,7 @@ class UploadService
end
def process!
media_file = Utils::get_file_for_upload(replacement.replacement_url, nil, replacement.replacement_file&.tempfile)
media_file = get_file_for_upload(replacement.replacement_url, nil, replacement.replacement_file&.tempfile)
if Post.where.not(id: post.id).exists?(md5: media_file.md5)
raise Error, "Duplicate: post with md5 #{media_file.md5} already exists"
@@ -69,5 +69,15 @@ class UploadService
note.rescale!(x_scale, y_scale)
end
end
def get_file_for_upload(source_url, referer_url, file)
return MediaFile.open(file) if file.present?
raise "No file or source URL provided" if source_url.blank?
strategy = Sources::Strategies.find(source_url, referer_url)
raise NotImplementedError, "No login credentials configured for #{strategy.site_name}." unless strategy.class.enabled?
strategy.download_file!
end
end
end

36
app/logical/upload_service/utils.rb
View File

@@ -1,36 +0,0 @@
# frozen_string_literal: true
class UploadService
module Utils
module_function
def is_downloadable?(source)
source =~ %r{\Ahttps?://}
end
def process_file(upload, file)
media_file = MediaFile.open(file)
upload.file = media_file
upload.file_ext = media_file.file_ext.to_s
upload.file_size = media_file.file_size
upload.md5 = media_file.md5
upload.image_width = media_file.width
upload.image_height = media_file.height
upload.validate!(:file)
MediaAsset.upload!(media_file)
end
def get_file_for_upload(source_url, referer_url, file)
return MediaFile.open(file) if file.present?
raise "No file or source URL provided" if source_url.blank?
strategy = Sources::Strategies.find(source_url, referer_url)
raise NotImplementedError, "No login credentials configured for #{strategy.site_name}." unless strategy.class.enabled?
strategy.download_file!
end
end
end