jaculabilis/danbooru
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

uploads: rework upload process.

Browse Source 
Rework the upload process so that files are saved to Danbooru first
before the user starts tagging the upload.

The main user-visible change is that you have to select the file first
before you can start tagging it. Saving the file first lets us fix a
number of problems:

* We can check for dupes before the user tags the upload.
* We can perform dupe checks and show preview images for users not using the bookmarklet.
* We can show preview images without having to proxy images through Danbooru.
* We can show previews of videos and ugoira files.
* We can reliably show the filesize and resolution of the image.
* We can let the user save files to upload later.
* We can get rid of a lot of spaghetti code related to preprocessing
  uploads. This was the cause of most weird "md5 confirmation doesn't
  match md5" errors.

(Not all of these are implemented yet.)

Internally, uploading is now a two-step process: first we create an upload
object, then we create a post from the upload. This is how it works:

* The user goes to /uploads/new and chooses a file or pastes an URL into
  the file upload component.
* The file upload component calls `POST /uploads` to create an upload.
* `POST /uploads` immediately returns a new upload object in the `pending` state.
* Danbooru starts processing the upload in a background job (downloading,
  resizing, and transferring the image to the image servers).
* The file upload component polls `/uploads/$id.json`, checking the
  upload `status` until it returns `completed` or `error`.
* When the upload status is `completed`, the user is redirected to /uploads/$id.
* On the /uploads/$id page, the user can tag the upload and submit it.
* The upload form calls `POST /posts` to create a new post from the upload.
* The user is redirected to the new post.

This is the data model:

* An upload represents a set of files uploaded to Danbooru by a user.
  Uploaded files don't have to belong to a post. An upload has an
  uploader, a status (pending, processing, completed, or error), a
  source (unless uploading from a file), and a list of media assets
  (image or video files).

* There is a has-and-belongs-to-many relationship between uploads and
  media assets. An upload can have many media assets, and a media asset
  can belong to multiple uploads. Uploads are joined to media assets
  through a upload_media_assets table.

  An upload could potentially have multiple media assets if it's a Pixiv
  or Twitter gallery. This is not yet implemented (at the moment all
  uploads have one media asset).

  A media asset can belong to multiple uploads if multiple people try
  to upload the same file, or if the same user tries to upload the same
  file more than once.

New features:

* On the upload page, you can press Ctrl+V to paste an URL and immediately upload it.
* You can save files for upload later. Your saved files are at /uploads.

Fixes:

* Improved error messages when uploading invalid files, bad URLs, and
  when forgetting the rating.
This commit is contained in:
evazion
2022-01-26 00:27:47 -06:00
parent f11c46b4f8
commit abdab7a0a8
46 changed files with 621 additions and 1016 deletions
Download Patch File Download Diff File Expand all files Collapse all files

21
app/components/file_upload_component.rb Normal file
View File

@@ -0,0 +1,21 @@
# frozen_string_literal: true
# A component for uploading files to Danbooru. Used on the /uploads/new page.
class FileUploadComponent < ApplicationComponent
attr_reader :url, :referer_url, :drop_target, :max_file_size
# @param url [String] Optional. The URL to upload. If present, the URL field
# will be prefilled in the widget and the upload will be immediately triggered.
# @param referer_url [String] Optional. The referrer URL passed by the bookmarklet.
# @param drop_target [String] A CSS selector. The target for drag and drop
# events. If "body", then files can be dropped anywhere on the page, not
# just on the upload widget itself.
# @param max_file_size [Integer] The max size in bytes of an upload.
def initialize(url: nil, referer_url: nil, drop_target: nil, max_file_size: Danbooru.config.max_file_size)
@url = url
@referer_url = referer_url
@drop_target = drop_target
@max_file_size = max_file_size
super
end
end

41
app/components/file_upload_component/file_upload_component.html.erb Normal file
View File

@@ -0,0 +1,41 @@
<div class="file-upload-component relative card w-md max-w-full" data-drop-target="<%= j drop_target %>" data-max-file-size="<%= j max_file_size %>">
<%= simple_form_for(Upload.new, url: uploads_path(format: :json), html: { class: "flex flex-col", autocomplete: "off" }, remote: true) do |f| %>
<%= f.input :file, as: :file, wrapper_html: { class: "hidden" } %>
<div class="dropzone-container input flex flex-col text-center items-center justify-center rounded-t-lg cursor-pointer">
<div class="dropzone-hint py-8">
<div>Choose file or drag image here</div>
<div class="hint">Max size: <%= number_to_human_size(Danbooru.config.max_file_size) %>.</div>
</div>
</div>
<p class="text-center">&mdash; or &mdash;</p>
<%= f.input :source, label: false, as: :string, placeholder: "Paste URL here", input_html: { value: url, class: "text-center" }, wrapper_html: { class: "px-4 text-center" } %>
<%= f.input :referer_url, as: :hidden, input_html: { value: referer_url } %>
<%= f.submit "Upload", class: "button-primary text-center mx-auto hidden", "data-disable-with": false %>
<progress class="w-full hidden"></progress>
<% end %>
<template class="dropzone-preview-template">
<div class="dz-preview dz-file-preview flex flex-col text-center space-y-4 pt-8">
<img class="object-contain px-8 max-h-360px max-w-full" data-dz-thumbnail>
<div class="dz-details">
<div class="dz-filename">
<span data-dz-name></span>
</div>
<div class="dz-size" data-dz-size></div>
</div>
<div class="dz-progress absolute w-full h-1">
<div class="dz-upload h-1" data-dz-uploadprogress></div>
</div>
<div class="dz-error-message">
<p data-dz-errormessage></p>
</div>
</div>
</template>
</div>

139
app/components/file_upload_component/file_upload_component.js Normal file
View File

@@ -0,0 +1,139 @@
import Dropzone from 'dropzone';
import Utility from "../../javascript/src/javascripts/utility.js";
import capitalize from "lodash/capitalize";
export default class FileUploadComponent {
static initialize() {
$(".file-upload-component").toArray().forEach(element => {
new FileUploadComponent($(element));
});
}
constructor($component) {
this.$component = $component;
this.$component.on("ajax:success", e => this.onSubmit(e));
this.$component.on("ajax:error", e => this.onError(e));
this.$dropTarget.on("paste.danbooru", e => this.onPaste(e));
this.dropzone = this.initializeDropzone();
// If the source field is pre-filled, then immediately submit the upload.
if (/^https?:\/\//.test(this.$sourceField.val())) {
this.$component.find("input[type='submit']").click();
}
}
initializeDropzone() {
if (!window.FileReader) {
this.$dropzone.addClass("hidden");
this.$component.find("input[type='file']").removeClass("hidden");
return null;
}
let dropzone = new Dropzone(this.$dropTarget.get(0), {
url: "/uploads.json",
paramName: "upload[file]",
clickable: this.$dropzone.get(0),
previewsContainer: this.$dropzone.get(0),
thumbnailHeight: null,
thumbnailWidth: null,
addRemoveLinks: false,
maxFiles: 1,
maxFilesize: this.maxFileSize,
maxThumbnailFilesize: this.maxFileSize,
timeout: 0,
acceptedFiles: "image/jpeg,image/png,image/gif,video/mp4,video/webm",
previewTemplate: this.$component.find(".dropzone-preview-template").html(),
});
dropzone.on("complete", file => {
this.$dropzone.find(".dz-progress").hide();
});
dropzone.on("addedfile", file => {
this.$dropzone.removeClass("error");
this.$dropzone.find(".dropzone-hint").hide();
// Remove all files except the file just added.
dropzone.files.forEach(f => {
if (f !== file) {
dropzone.removeFile(f);
}
});
});
dropzone.on("success", file => {
this.$dropzone.addClass("success");
let upload = JSON.parse(file.xhr.response)
location.href = `/uploads/${upload.id}`;
});
dropzone.on("error", (file, msg) => {
this.$dropzone.addClass("error");
});
return dropzone;
}
onPaste(e) {
let url = e.originalEvent.clipboardData.getData("text");
this.$component.find("input[name='upload[source]']:not([disabled])").val(url);
if (/^https?:\/\//.test(url)) {
this.$component.find("input[type='submit']:not([disabled])").click();
}
}
// Called after the upload is submitted via AJAX. Polls the upload until it
// is complete, then redirects to the upload page.
async onSubmit(e) {
let upload = e.originalEvent.detail[0];
this.$component.find("progress").removeClass("hidden");
this.$component.find("input").attr("disabled", "disabled");
while (upload.status === "pending" || upload.status === "processing") {
await Utility.delay(500);
upload = await $.get(`/uploads/${upload.id}.json`);
}
if (upload.status === "completed") {
location.href = `/uploads/${upload.id}`;
} else {
this.$component.find("progress").addClass("hidden");
this.$component.find("input").removeAttr("disabled");
Utility.error(upload.status);
}
}
// Called when creating the upload failed because of a validation error (normally, because the source URL was not a real URL).
async onError(e) {
let errors = e.originalEvent.detail[0].errors;
let message = Object.keys(errors).map(attribute => {
return errors[attribute].map(error => `${capitalize(attribute)} ${error}`);
}).join("; ");
Utility.error(message);
}
get $dropzone() {
return this.$component.find(".dropzone-container");
}
get $sourceField() {
return this.$component.find("input[name='upload[source]']");
}
get maxFileSize() {
return Number(this.$component.attr("data-max-file-size")) / (1024 * 1024);
}
// The element to listen for drag and drop events and paste events. By default,
// it's the `.file-upload-component` element. If `data-drop-target` is the `body`
// element, then you can drop images or paste URLs anywhere on the page.
get $dropTarget() {
return $(this.$component.attr("data-drop-target") || this.$component);
}
}
$(FileUploadComponent.initialize);

32
app/components/file_upload_component/file_upload_component.scss Normal file
View File

@@ -0,0 +1,32 @@
.file-upload-component {
form {
margin: 0;
}
progress {
height: 6px;
}
.dropzone-container {
background: var(--uploads-dropzone-background);
&.error {
background: var(--error-background-color);
}
&.success {
background: var(--success-background-color);
}
}
.dz-progress {
bottom: 0;
left: 0;
background-color: var(--uploads-dropzone-progress-bar-background-color);
.dz-upload {
background-color: var(--uploads-dropzone-progress-bar-foreground-color);
}
}
}

2
app/components/source_data_component/source_data_component.html.erb
View File

@@ -1,4 +1,4 @@
<div class="source-data p-4 mb-4">
<div class="source-data p-4 mt-4 mb-4">
<%= link_to "Fetch source data", source_path, class: "source-data-fetch" %>
<%= spinner_icon class: "source-data-loading" %>

5
app/components/source_data_component/source_data_component.js
View File

@@ -1,12 +1,11 @@
class SourceDataComponent {
static initialize() {
$(document).on("change.danbooru", "#upload_source", SourceDataComponent.fetchData);
$(document).on("click.danbooru", ".source-data-fetch", SourceDataComponent.fetchData);
}
static async fetchData(e) {
let url = $("#upload_source,#post_source").val();
let ref = $("#upload_referer_url").val();
let url = $("#post_source").val();
let ref = $("#post_referer_url").val();
e.preventDefault();

13
app/controllers/posts_controller.rb
View File

@@ -63,6 +63,19 @@ class PostsController < ApplicationController
respond_with_post_after_update(@post)
end
def create
@post = authorize Post.new_from_upload(permitted_attributes(Post))
@post.save
if @post.errors.any?
@upload = UploadMediaAsset.find(params[:post][:upload_media_asset_id]).upload
flash[:notice] = @post.errors.full_messages.join("; ")
respond_with(@post, render: { template: "uploads/show" })
else
respond_with(@post)
end
end
def destroy
@post = authorize Post.find(params[:id])

42
app/controllers/uploads_controller.rb
View File

@@ -2,14 +2,16 @@
class UploadsController < ApplicationController
respond_to :html, :xml, :json, :js
skip_before_action :verify_authenticity_token, only: [:preprocess]
skip_before_action :verify_authenticity_token, only: [:create], if: -> { request.xhr? }
def new
authorize Upload
@source = Sources::Strategies.find(params[:url], params[:ref]) if params[:url].present?
@upload, @remote_size = UploadService::ControllerHelper.prepare(
url: params[:url], ref: params[:ref]
)
@upload = authorize Upload.new(uploader: CurrentUser.user, uploader_ip_addr: CurrentUser.ip_addr, rating: "q", tag_string: "", source: params[:url], referer_url: params[:ref], **permitted_attributes(Upload))
respond_with(@upload)
end
def create
@upload = authorize Upload.new(uploader: CurrentUser.user, uploader_ip_addr: CurrentUser.ip_addr, rating: "q", tag_string: "", **permitted_attributes(Upload))
@upload.save
respond_with(@upload)
end
@@ -28,38 +30,14 @@ class UploadsController < ApplicationController
def index
@uploads = authorize Upload.visible(CurrentUser.user).paginated_search(params, count_pages: true)
@uploads = @uploads.includes(:uploader, post: [:media_asset, :uploader]) if request.format.html?
@uploads = @uploads.includes(:uploader, :media_assets) if request.format.html?
respond_with(@uploads)
end
def show
@upload = authorize Upload.find(params[:id])
respond_with(@upload) do |format|
format.html do
if @upload.is_completed? && @upload.post_id
redirect_to(post_path(@upload.post_id))
end
end
end
end
def preprocess
authorize Upload
@upload, @remote_size = UploadService::ControllerHelper.prepare(
url: params.dig(:upload, :source), file: params.dig(:upload, :file), ref: params.dig(:upload, :referer_url)
)
render body: nil
end
def create
@service = authorize UploadService.new(permitted_attributes(Upload)), policy_class: UploadPolicy
@upload = @service.start!
if @service.warnings.any?
flash[:notice] = @service.warnings.join(".\n \n")
end
@post = Post.new(uploader: @upload.uploader, uploader_ip_addr: @upload.uploader_ip_addr, source: @upload.source, rating: nil, **permitted_attributes(Post))
respond_with(@upload)
end
end

19
app/helpers/uploads_helper.rb
View File

@@ -1,19 +0,0 @@
# frozen_string_literal: true
module UploadsHelper
def render_status(upload)
case upload.status
when /duplicate: (\d+)/
dup_post_id = $1
link_to(upload.status.gsub(/error: RuntimeError - /, ""), post_path(dup_post_id))
when /\Aerror: /
search_params = params[:search].permit!
link_to(upload.sanitized_status, uploads_path(search: search_params.merge(status: upload.sanitized_status)))
else
search_params = params[:search].permit!
link_to(upload.status, uploads_path(search: search_params.merge(status: upload.status)))
end
end
end

2
app/javascript/packs/application.js
View File

@@ -39,6 +39,7 @@ import CommentVotesTooltipComponent from "../../components/comment_votes_tooltip
import CurrentUser from "../src/javascripts/current_user.js";
import Dtext from "../src/javascripts/dtext.js";
import FavoritesTooltipComponent from "../../components/favorites_tooltip_component/favorites_tooltip_component.js";
import FileUploadComponent from "../../components/file_upload_component/file_upload_component.js";
import ForumPostComponent from "../../components/forum_post_component/forum_post_component.js";
import IqdbQuery from "../src/javascripts/iqdb_queries.js";
import Note from "../src/javascripts/notes.js";
@@ -64,6 +65,7 @@ Danbooru.CommentVotesTooltipComponent = CommentVotesTooltipComponent;
Danbooru.CurrentUser = CurrentUser;
Danbooru.Dtext = Dtext;
Danbooru.FavoritesTooltipComponent = FavoritesTooltipComponent;
Danbooru.FileUploadComponent = FileUploadComponent;
Danbooru.ForumPostComponent = ForumPostComponent;
Danbooru.IqdbQuery = IqdbQuery;
Danbooru.Note = Note;

8
app/javascript/src/javascripts/posts.js
View File

@@ -39,7 +39,7 @@ Post.initialize_all = function() {
this.initialize_ruffle_player();
}
if ($("#c-posts #a-show, #c-uploads #a-new").length) {
if ($("#c-posts #a-show, #c-uploads #a-show").length) {
this.initialize_edit_dialog();
}
@@ -102,7 +102,7 @@ Post.open_edit_dialog = function() {
$("#post-sections li").removeClass("active");
$("#post-edit-link").parent("li").addClass("active");
var $tag_string = $("#post_tag_string,#upload_tag_string");
var $tag_string = $("#post_tag_string");
$("#open-edit-dialog").hide();
var dialog = $("<div/>").attr("id", "edit-dialog");
@@ -157,9 +157,9 @@ Post.open_edit_dialog = function() {
}
Post.close_edit_dialog = function(e, ui) {
$("#form").appendTo($("#c-posts #edit,#c-uploads #a-new"));
$("#form").appendTo($("#c-posts #edit,#c-uploads #a-show"));
$("#edit-dialog").remove();
var $tag_string = $("#post_tag_string,#upload_tag_string");
var $tag_string = $("#post_tag_string");
$("div.input").has($tag_string).prevAll().show();
$("#open-edit-dialog").show();
$tag_string.css({"resize": "", "width": ""});

10
app/javascript/src/javascripts/related_tag.js
View File

@@ -9,7 +9,7 @@ RelatedTag.initialize_all = function() {
$(document).on("click.danbooru", ".related-tags a", RelatedTag.toggle_tag);
$(document).on("click.danbooru", "#show-related-tags-link", RelatedTag.show);
$(document).on("click.danbooru", "#hide-related-tags-link", RelatedTag.hide);
$(document).on("keyup.danbooru.relatedTags", "#upload_tag_string, #post_tag_string", RelatedTag.update_selected);
$(document).on("keyup.danbooru.relatedTags", "#post_tag_string", RelatedTag.update_selected);
$(document).on("danbooru:update-source-data", RelatedTag.on_update_source_data);
$(document).on("danbooru:open-post-edit-dialog", RelatedTag.hide);
@@ -21,7 +21,7 @@ RelatedTag.initialize_all = function() {
// Show the related tags automatically when the "Edit" tab is opened, or by default on the uploads page.
$(document).on("danbooru:open-post-edit-tab", RelatedTag.show);
if ($("#c-uploads #a-new").length) {
if ($("#c-uploads #a-show").length) {
RelatedTag.show();
}
}
@@ -50,7 +50,7 @@ RelatedTag.current_tag = function() {
// 7. |abc def -> abc
// 8. | abc def -> abc
var $field = $("#upload_tag_string,#post_tag_string");
var $field = $("#post_tag_string");
var string = $field.val();
var n = string.length;
var a = $field.prop('selectionStart');
@@ -105,12 +105,12 @@ RelatedTag.update_selected = function(e) {
}
RelatedTag.current_tags = function() {
let tagString = $("#upload_tag_string,#post_tag_string").val().toLowerCase();
let tagString = $("#post_tag_string").val().toLowerCase();
return Utility.splitWords(tagString);
}
RelatedTag.toggle_tag = function(e) {
var $field = $("#upload_tag_string,#post_tag_string");
var $field = $("#post_tag_string");
var tag = $(e.target).closest("li").text().trim().replace(/ /g, "_");
if (RelatedTag.current_tags().includes(tag)) {

158
app/javascript/src/javascripts/uploads.js
View File

@@ -1,6 +1,3 @@
import Dropzone from 'dropzone';
import SparkMD5 from 'spark-md5';
let Upload = {};
Upload.IQDB_LIMIT = 5;
@@ -8,11 +5,9 @@ Upload.IQDB_MIN_SIMILARITY = 50;
Upload.IQDB_HIGH_SIMILARITY = 70;
Upload.initialize_all = function() {
if ($("#c-uploads").length) {
if ($("#c-uploads #a-show").length) {
this.initialize_image();
this.initialize_similar();
this.initialize_submit();
$("#similar-button").click();
$("#toggle-artist-commentary").on("click.danbooru", function(e) {
Upload.toggle_commentary();
@@ -23,104 +18,57 @@ Upload.initialize_all = function() {
Upload.toggle_translation();
e.preventDefault();
});
}
if ($("#c-uploads #a-batch").length) {
$(document).on("click.danbooru", "#c-uploads #a-batch #link", Upload.batch_open_all);
}
if ($("#c-uploads #a-new").length) {
this.initialize_dropzone();
}
}
Upload.initialize_submit = function() {
$("#form").on("submit.danbooru", Upload.validate_upload);
}
Upload.validate_upload = function (e) {
var error_messages = [];
if (($("#upload_file").val() === undefined) && !/^https?:\/\/.+/i.test($("#upload_source").val()) && $("#upload_md5_confirmation").val() === "") {
error_messages.push("Must choose file or specify source");
} else if ($(".dz-progress:visible").length) {
error_messages.push("File has not finished uploading yet")
}
if (!$("#upload_rating_s").prop("checked") && !$("#upload_rating_q").prop("checked") && !$("#upload_rating_e").prop("checked") &&
($("#upload_tag_string").val().search(/\brating:[sqe]/i) < 0)) {
error_messages.push("Must specify a rating");
}
if (error_messages.length === 0) {
$("#submit-button").prop("disabled", "true");
$("#submit-button").prop("value", "Submitting...");
$("#client-errors").hide();
} else {
$("#client-errors").html("<strong>Error</strong>: " + error_messages.join(", "));
$("#client-errors").show();
e.preventDefault();
}
}
Upload.initialize_similar = function() {
$("#similar-button").on("click.danbooru", function(e) {
e.preventDefault();
let source = $("#post_source").val();
let source = $("#upload_source").val();
if (/^https?:\/\//.test(source)) {
$.get("/iqdb_queries.js", {
limit: Upload.IQDB_LIMIT,
search: {
url: source,
similarity: Upload.IQDB_MIN_SIMILARITY,
high_similarity: Upload.IQDB_HIGH_SIMILARITY
}
});
}
});
if (/^https?:\/\//.test(source)) {
$.get("/iqdb_queries.js", {
limit: Upload.IQDB_LIMIT,
search: {
url: source,
similarity: Upload.IQDB_MIN_SIMILARITY,
high_similarity: Upload.IQDB_HIGH_SIMILARITY
}
});
}
}
Upload.initialize_image = function() {
let $image = $("#image");
if ($image.prop("complete")) {
Upload.update_scale();
} else {
$image.on("load.danbooru", Upload.update_scale);
}
$(window).on("resize.danbooru", Upload.update_scale);
$(document).on("click.danbooru", "#image", Upload.toggle_size);
$(document).on("click.danbooru", "#upload-image-view-small", Upload.view_small);
$(document).on("click.danbooru", "#upload-image-view-large", Upload.view_large);
$(document).on("click.danbooru", "#upload-image-view-full", Upload.view_full);
}
Upload.no_image_available = function(e) {
$("#a-new").addClass("no-image-available");
}
Upload.view_small = function(e) {
$("#image").addClass("fit-width fit-height");
$("#a-new").attr("data-image-size", "small");
Upload.update_scale();
$("#a-show").attr("data-image-size", "small");
e.preventDefault();
}
Upload.view_large = function(e) {
$("#image").removeClass("fit-height").addClass("fit-width");
$("#a-new").attr("data-image-size", "large");
Upload.update_scale();
$("#a-show").attr("data-image-size", "large");
e.preventDefault();
}
Upload.view_full = function(e) {
$("#image").removeClass("fit-width fit-height");
$("#a-new").attr("data-image-size", "full");
Upload.update_scale();
$("#a-show").attr("data-image-size", "full");
e.preventDefault();
}
Upload.toggle_size = function(e) {
let window_aspect_ratio = $(window).width() / $(window).height();
let image_aspect_ratio = $("#image").width() / $("#image").height();
let image_size = $("#a-new").attr("data-image-size");
let image_size = $("#a-show").attr("data-image-size");
if (image_size === "small" && image_aspect_ratio >= window_aspect_ratio) {
Upload.view_full(e);
@@ -133,17 +81,6 @@ Upload.toggle_size = function(e) {
}
}
Upload.update_scale = function() {
let $image = $("#image");
if ($image.length) {
let natural_width = $image.get(0).naturalWidth;
let natural_height = $image.get(0).naturalHeight;
let scale_percentage = Math.round(100 * $image.width() / natural_width);
$("#upload-image-metadata-resolution").html(`(${natural_width}x${natural_height}, resized to ${scale_percentage}%)`);
}
}
Upload.toggle_commentary = function() {
if ($(".artist-commentary").is(":visible")) {
$("#toggle-artist-commentary").text("show »");
@@ -165,69 +102,10 @@ Upload.toggle_translation = function() {
$(".commentary-translation").slideToggle();
};
Upload.initialize_dropzone = function() {
if (!window.FileReader) {
$("#filedropzone").remove();
return;
}
let dropzone = new Dropzone(document.body, {
paramName: "upload[file]",
url: "/uploads/preprocess",
clickable: "#filedropzone",
previewsContainer: "#filedropzone",
thumbnailHeight: 150,
thumbnailWidth: 150,
thumbnailMethod: "contain",
addRemoveLinks: false,
maxFiles: 1,
maxFilesize: Upload.max_file_size(),
maxThumbnailFilesize: Upload.max_file_size(),
timeout: 0,
acceptedFiles: "image/jpeg,image/png,image/gif,video/mp4,video/webm",
previewTemplate: $("#dropzone-preview-template").html(),
init: function() {
$(".fallback").hide();
this.on("complete", function(file) {
$("#filedropzone .dz-progress").hide();
});
this.on("addedfile", function(file) {
$("#filedropzone .dropzone-hint").hide();
// replace the previous file with the new one.
dropzone.files.forEach(f => {
if (f !== file) {
dropzone.removeFile(f);
}
});
let reader = new FileReader();
reader.addEventListener("loadend", function() {
let buf = new SparkMD5.ArrayBuffer();
buf.append(this.result);
let hash = buf.end();
$("#upload_md5_confirmation").val(hash);
});
reader.readAsArrayBuffer(file);
});
this.on("success", function(file) {
$("#filedropzone").addClass("success");
});
this.on("error", function(file, msg) {
$("#filedropzone").addClass("error");
});
}
});
};
Upload.batch_open_all = function() {
$(".upload-preview > a").each((_i, link) => window.open(link.href));
};
Upload.max_file_size = function() {
return Number($("meta[name=max-file-size]").attr("content")) / (1024 * 1024);
};
$(function() {
Upload.initialize_all();
});

41
app/javascript/src/styles/common/utilities.scss
View File

@@ -12,6 +12,8 @@ $spacer: 0.25rem; /* 4px */
.font-monospace { font: var(--monospace-font); }
.font-bold { font-weight: bold; }
.cursor-pointer { cursor: pointer; }
.hidden { display: none !important; }
.inline-block { display: inline-block; }
.block { display: block; }
@@ -36,14 +38,27 @@ $spacer: 0.25rem; /* 4px */
.leading-none { line-height: 1; }
.absolute { position: absolute; }
.relative { position: relative; }
.top-0\.5 { top: 0.5 * $spacer; }
.bottom-0\.5 { bottom: 0.5 * $spacer; }
.left-0\.5 { left: 0.5 * $spacer; }
.right-0\.5 { right: 0.5 * $spacer; }
.rounded-sm { border-radius: 0.5 * $spacer; }
.rounded { border-radius: 1 * $spacer; }
.border, %border { border-width: 1px; }
.rounded-sm, %rounded-sm { border-radius: 0.5 * $spacer; }
.rounded, %rounded { border-radius: 1 * $spacer; }
.rounded-lg, %rounded-lg { border-radius: 2 * $spacer; }
.rounded-t-sm, %rounded-t-sm { border-top-left-radius: 0.5 * $spacer; border-top-right-radius: 0.5 * $spacer; }
.rounded-t, %rounded-t { border-top-left-radius: 1 * $spacer; border-top-right-radius: 1 * $spacer; }
.rounded-t-lg, %rounded-t-lg { border-top-left-radius: 2 * $spacer; border-top-right-radius: 2 * $spacer; }
.shadow-md, %shadow-md { box-shadow: 0 4px 6px -1px rgb(0 0 0 / 0.1), 0 2px 4px -2px rgb(0 0 0 / 0.1); }
.shadow-lg, %shadow-lg { box-shadow: 0 10px 15px -3px rgb(0 0 0 / 0.1), 0 4px 6px -4px rgb(0 0 0 / 0.1); }
.object-contain { object-fit: contain; }
.m-0 { margin: 0; }
.m-px { margin: 1px; }
@@ -69,15 +84,31 @@ $spacer: 0.25rem; /* 4px */
.p-0\.5 { padding: 0.5 * $spacer; }
.p-4 { padding: 4 * $spacer; }
.px-4 { padding-left: 4 * $spacer; padding-right: 4 * $spacer; }
.px-8 { padding-left: 8 * $spacer; padding-right: 8 * $spacer; }
.py-4 { padding-top: 4 * $spacer; padding-bottom: 4 * $spacer; }
.py-8 { padding-top: 8 * $spacer; padding-bottom: 8 * $spacer; }
.pt-4 { padding-top: 4 * $spacer; }
.pt-8 { padding-top: 8 * $spacer; }
.pr-2 { padding-right: 2 * $spacer; }
.pr-4 { padding-right: 4 * $spacer; }
.w-sm { width: 24rem; }
.w-md { width: 28rem; }
.w-1\/4 { width: 25%; }
.w-full { width: 100%; }
.max-w-full { max-width: 100%; }
.h-1 { height: 1 * $spacer; }
.h-3 { height: 3 * $spacer; }
.h-10 { height: 10 * $spacer; }
.max-h-360px { max-height: 360px; }
.space-x-1 > * + * { margin-left: 1 * $spacer; }
.space-x-2 > * + * { margin-left: 2 * $spacer; }
.space-x-4 > * + * { margin-left: 4 * $spacer; }
@@ -111,6 +142,12 @@ $spacer: 0.25rem; /* 4px */
.grid-cols-8 { grid-template-columns: repeat(8, minmax(0, 1fr)); }
.grid-cols-12 { grid-template-columns: repeat(12, minmax(0, 1fr)); }
.card {
@extend %border;
@extend %rounded-lg;
@extend %shadow-md;
}
.thin-scrollbar {
overflow-x: hidden;
overflow-y: auto;

59
app/javascript/src/styles/specific/dropzone.scss
View File

@@ -1,59 +0,0 @@
#filedropzone {
background: var(--uploads-dropzone-background);
padding: 0;
min-height: 100px;
display: flex;
flex-direction: column;
align-items: center;
justify-content: center;
border-radius: 4px;
cursor: pointer;
position: relative;
.dz-preview {
display: flex;
flex-direction: column;
text-align: center;
&.dz-image-preview img {
margin: 1em 0;
object-fit: contain;
}
.dz-details {
margin-bottom: 1em;
.dz-filename, .dz-size {
display: inline;
}
}
}
&.dz-started .dropzone-hint {
display: none;
}
&.error {
background: var(--error-background-color);
}
&.success {
background: var(--success-background-color);
}
}
.dz-progress {
position: absolute;
width: 100%;
bottom: 0;
left: 0;
height: 5px;
background-color: var(--uploads-dropzone-progress-bar-background-color);
.dz-upload {
background-color: var(--uploads-dropzone-progress-bar-foreground-color);
display: block;
height: 5px;
}
}

2
app/javascript/src/styles/specific/posts.scss
View File

@@ -25,7 +25,7 @@
#edit-dialog {
/* Hide everything but the rating and tags fields. */
.post_has_embedded_notes_fieldset, .post_lock_fieldset, .post_parent_id,
.post_source, #filedropzone, .upload_as_pending, .upload_source_container,
.post_source, .dropzone-container, .upload_as_pending, .upload_source_container,
.upload_parent_id, .upload_artist_commentary_container, .upload_commentary_translation_container {
display: none;
}

20
app/javascript/src/styles/specific/uploads.scss
View File

@@ -1,19 +1,5 @@
div#c-uploads {
div#a-new {
#no-image-available {
display: none;
}
&.no-image-available {
#upload-image-metadata, #image, #iqdb-similar {
display: none;
}
#no-image-available {
display: block !important;
}
}
div#a-show {
&[data-image-size="small"] {
#image {
cursor: zoom-in;
@@ -74,10 +60,6 @@ div#c-uploads {
}
}
textarea#upload_tag_string {
width: 100%;
}
div.field_with_errors {
display: inline;
}

9
app/jobs/process_upload_job.rb Normal file
View File

@@ -0,0 +1,9 @@
# frozen_string_literal: true
class ProcessUploadJob < ApplicationJob
queue_with_priority -1
def perform(upload)
upload.process_upload!
end
end

12
app/jobs/upload_preprocessor_delayed_start_job.rb
View File

@@ -1,12 +0,0 @@
# frozen_string_literal: true
# A job that downloads and generates thumbnails in the background for an image
# uploaded with the upload bookmarklet.
class UploadPreprocessorDelayedStartJob < ApplicationJob
queue_as :default
queue_with_priority(-1)
def perform(source, referer_url, uploader)
UploadService::Preprocessor.new(source: source, referer_url: referer_url).delayed_start(uploader)
end
end

11
app/jobs/upload_service_delayed_start_job.rb
View File

@@ -1,11 +0,0 @@
# frozen_string_literal: true
# A job that tries to resume a preprocessed image upload job.
class UploadServiceDelayedStartJob < ApplicationJob
queue_as :default
queue_with_priority(-1)
def perform(params, uploader)
UploadService.new(params).delayed_start(uploader)
end
end

9
app/logical/sources/strategies/base.rb
View File

@@ -214,15 +214,6 @@ module Sources
{}
end
# Returns the size of the image resource without actually downloading the file.
def remote_size
response = http_downloader.head(image_url)
return nil unless response.status == 200 && response.content_length.present?
response.content_length.to_i
end
memoize :remote_size
# Download the file at the given url, or at the main image url by default.
def download_file!(download_url = image_url)
raise DownloadError, "Download failed: couldn't find download url for #{url}" if download_url.blank?

106
app/logical/upload_service.rb
View File

@@ -1,106 +0,0 @@
# frozen_string_literal: true
# A service object for uploading an image.
class UploadService
attr_reader :params, :post, :upload
def initialize(params)
@params = params
end
def delayed_start(uploader)
CurrentUser.scoped(uploader) do
start!
end
rescue ActiveRecord::RecordNotUnique
end
def start!
preprocessor = Preprocessor.new(params)
if preprocessor.in_progress?
UploadServiceDelayedStartJob.set(wait: 5.seconds).perform_later(params, CurrentUser.user)
return preprocessor.predecessor
end
if preprocessor.completed?
@upload = preprocessor.finish!
begin
create_post_from_upload(@upload)
rescue Exception => e
@upload.update(status: "error: #{e.class} - #{e.message}", backtrace: e.backtrace.join("\n"))
end
return @upload
end
params[:rating] ||= "q"
params[:tag_string] ||= "tagme"
@upload = Upload.create!(params)
begin
if @upload.invalid?
return @upload
end
@upload.update(status: "processing")
@upload.file = Utils.get_file_for_upload(@upload.source_url, @upload.referer_url, @upload.file&.tempfile)
Utils.process_file(upload, @upload.file)
@upload.save!
@post = create_post_from_upload(@upload)
@upload
rescue Exception => e
@upload.update(status: "error: #{e.class} - #{e.message}", backtrace: e.backtrace.join("\n"))
@upload
end
end
def warnings
return [] if @post.nil?
@post.warnings.full_messages
end
def create_post_from_upload(upload)
@post = convert_to_post(upload)
@post.save!
if upload.has_commentary?
@post.create_artist_commentary(
:original_title => upload.artist_commentary_title,
:original_description => upload.artist_commentary_desc,
:translated_title => upload.translated_commentary_title,
:translated_description => upload.translated_commentary_desc
)
end
@post.update_iqdb
upload.update(status: "completed", post_id: @post.id)
@post
end
def convert_to_post(upload)
Post.new.tap do |p|
p.tag_string = upload.tag_string
p.md5 = upload.md5
p.file_ext = upload.file_ext
p.image_width = upload.image_width
p.image_height = upload.image_height
p.rating = upload.rating
if upload.source.present?
p.source = Sources::Strategies.find(upload.source, upload.referer_url).canonical_url || upload.source
end
p.file_size = upload.file_size
p.uploader_id = upload.uploader_id
p.uploader_ip_addr = upload.uploader_ip_addr
p.parent_id = upload.parent_id
if !upload.uploader.can_upload_free? || upload.upload_as_pending?
p.is_pending = true
end
end
end
end

26
app/logical/upload_service/controller_helper.rb
View File

@@ -1,26 +0,0 @@
# frozen_string_literal: true
class UploadService
module ControllerHelper
def self.prepare(url: nil, file: nil, ref: nil)
upload = Upload.new
if Utils.is_downloadable?(url) && file.nil?
# this gets called from UploadsController#new so we need to preprocess async
UploadPreprocessorDelayedStartJob.perform_later(url, ref, CurrentUser.user)
strategy = Sources::Strategies.find(url, ref)
remote_size = strategy.remote_size
return [upload, remote_size]
end
if file
# this gets called via XHR so we can process sync
Preprocessor.new(file: file).delayed_start(CurrentUser.user)
end
[upload]
end
end
end

95
app/logical/upload_service/preprocessor.rb
View File

@@ -1,95 +0,0 @@
# frozen_string_literal: true
class UploadService
class Preprocessor
extend Memoist
attr_reader :params
def initialize(params)
@params = params
end
def source
params[:source]
end
def md5
params[:md5_confirmation]
end
def referer_url
params[:referer_url]
end
def in_progress?
if md5.present?
Upload.exists?(status: "preprocessing", md5: md5)
elsif Utils.is_downloadable?(source)
Upload.exists?(status: "preprocessing", source: source)
else
false
end
end
def predecessor
if md5.present?
Upload.where(status: ["preprocessed", "preprocessing"], md5: md5).first
elsif Utils.is_downloadable?(source)
Upload.where(status: ["preprocessed", "preprocessing"], source: source).first
end
end
def completed?
predecessor.present?
end
def delayed_start(uploader)
CurrentUser.scoped(uploader) do
start!
end
rescue ActiveRecord::RecordNotUnique
end
def start!
params[:rating] ||= "q"
params[:tag_string] ||= "tagme"
upload = Upload.create!(params)
begin
upload.update(status: "preprocessing")
file = Utils.get_file_for_upload(upload.source_url, upload.referer_url, params[:file]&.tempfile)
Utils.process_file(upload, file)
upload.rating = params[:rating]
upload.tag_string = params[:tag_string]
upload.status = "preprocessed"
upload.save!
rescue Exception => e
upload.update(file_ext: nil, status: "error: #{e.class} - #{e.message}", backtrace: e.backtrace.join("\n"))
end
upload
end
def finish!(upload = nil)
pred = upload || predecessor
# regardless of who initialized the upload, credit should
# goto whoever submitted the form
pred.initialize_attributes
pred.attributes = params
# if a file was uploaded after the preprocessing occurred,
# then process the file and overwrite whatever the preprocessor
# did
Utils.process_file(pred, pred.file.tempfile) if pred.file.present?
pred.status = "completed"
pred.save
pred
end
end
end

12
app/logical/upload_service/replacer.rb
View File

@@ -26,7 +26,7 @@ class UploadService
end
def process!
media_file = Utils::get_file_for_upload(replacement.replacement_url, nil, replacement.replacement_file&.tempfile)
media_file = get_file_for_upload(replacement.replacement_url, nil, replacement.replacement_file&.tempfile)
if Post.where.not(id: post.id).exists?(md5: media_file.md5)
raise Error, "Duplicate: post with md5 #{media_file.md5} already exists"
@@ -69,5 +69,15 @@ class UploadService
note.rescale!(x_scale, y_scale)
end
end
def get_file_for_upload(source_url, referer_url, file)
return MediaFile.open(file) if file.present?
raise "No file or source URL provided" if source_url.blank?
strategy = Sources::Strategies.find(source_url, referer_url)
raise NotImplementedError, "No login credentials configured for #{strategy.site_name}." unless strategy.class.enabled?
strategy.download_file!
end
end
end

36
app/logical/upload_service/utils.rb
View File

@@ -1,36 +0,0 @@
# frozen_string_literal: true
class UploadService
module Utils
module_function
def is_downloadable?(source)
source =~ %r{\Ahttps?://}
end
def process_file(upload, file)
media_file = MediaFile.open(file)
upload.file = media_file
upload.file_ext = media_file.file_ext.to_s
upload.file_size = media_file.file_size
upload.md5 = media_file.md5
upload.image_width = media_file.width
upload.image_height = media_file.height
upload.validate!(:file)
MediaAsset.upload!(media_file)
end
def get_file_for_upload(source_url, referer_url, file)
return MediaFile.open(file) if file.present?
raise "No file or source URL provided" if source_url.blank?
strategy = Sources::Strategies.find(source_url, referer_url)
raise NotImplementedError, "No login credentials configured for #{strategy.site_name}." unless strategy.class.enabled?
strategy.download_file!
end
end
end

23
app/models/media_asset.rb
View File

@@ -4,6 +4,10 @@ class MediaAsset < ApplicationRecord
class Error < StandardError; end
VARIANTS = %i[preview 180x180 360x360 720x720 sample original]
MAX_VIDEO_DURATION = 140 # 2:20
MAX_IMAGE_RESOLUTION = Danbooru.config.max_image_resolution
MAX_IMAGE_WIDTH = Danbooru.config.max_image_width
MAX_IMAGE_HEIGHT = Danbooru.config.max_image_height
ENABLE_SEO_POST_URLS = Danbooru.config.enable_seo_post_urls
LARGE_IMAGE_WIDTH = Danbooru.config.large_image_width
STORAGE_SERVICE = Danbooru.config.storage_manager
@@ -30,6 +34,9 @@ class MediaAsset < ApplicationRecord
}
validates :md5, uniqueness: { conditions: -> { where(status: [:processing, :active]) } }
validates :file_ext, inclusion: { in: %w[jpg png gif mp4 webm swf zip], message: "Not an image or video" }
validates :duration, numericality: { less_than_or_equal_to: MAX_VIDEO_DURATION, message: "must be less than #{MAX_VIDEO_DURATION} seconds", allow_nil: true }, on: :create # XXX should allow admins to bypass
validate :validate_resolution, on: :create
class Variant
extend Memoist
@@ -194,6 +201,8 @@ class MediaAsset < ApplicationRecord
# This can't be called inside a transaction because the transaction will
# fail if there's a RecordNotUnique error when the asset already exists.
def upload!(media_file)
raise Error, "File is corrupt" if media_file.is_corrupt?
media_asset = create!(file: media_file, status: :processing)
media_asset.distribute_files!(media_file)
media_asset.update!(status: :active)
@@ -318,4 +327,18 @@ class MediaAsset < ApplicationRecord
is_animated? && file_ext == "png"
end
end
concerning :ValidationMethods do
def validate_resolution
resolution = image_width * image_height
if resolution > MAX_IMAGE_RESOLUTION
errors.add(:base, "Image resolution is too large (resolution: #{(resolution / 1_000_000.0).round(1)} megapixels (#{image_width}x#{image_height}); max: #{MAX_IMAGE_RESOLUTION / 1_000_000} megapixels)")
elsif image_width > MAX_IMAGE_WIDTH
errors.add(:image_width, "is too large (width: #{image_width}; max width: #{MAX_IMAGE_WIDTH})")
elsif image_height > MAX_IMAGE_HEIGHT
errors.add(:image_height, "is too large (height: #{image_height}; max height: #{MAX_IMAGE_HEIGHT})")
end
end
end
end

36
app/models/post.rb
View File

@@ -18,8 +18,9 @@ class Post < ApplicationRecord
before_validation :parse_pixiv_id
before_validation :blank_out_nonexistent_parents
before_validation :remove_parent_loops
validates :md5, uniqueness: { message: ->(post, _data) { "duplicate: #{Post.find_by_md5(post.md5).id}" }}, on: :create
validates :rating, inclusion: { in: %w[s q e], message: "rating must be s, q, or e" }
validates :md5, uniqueness: { message: ->(post, _data) { "Duplicate of post ##{Post.find_by_md5(post.md5).id}" }}, on: :create
validates :rating, presence: { message: "not selected" }
validates :rating, inclusion: { in: %w[s q e], message: "must be S, Q, or E" }, if: -> { rating.present? }
validates :source, length: { maximum: 1200 }
validate :added_tags_are_valid
validate :removed_tags_are_valid
@@ -34,6 +35,7 @@ class Post < ApplicationRecord
after_save :create_version
after_save :update_parent_on_save
after_save :apply_post_metatags
after_create_commit :update_iqdb
belongs_to :approver, class_name: "User", optional: true
belongs_to :uploader, :class_name => "User", :counter_cache => "post_upload_count"
@@ -73,6 +75,36 @@ class Post < ApplicationRecord
has_many :versions, -> { Rails.env.test? ? order("post_versions.updated_at ASC, post_versions.id ASC") : order("post_versions.updated_at ASC") }, class_name: "PostVersion", dependent: :destroy
end
def self.new_from_upload(params)
upload_media_asset = UploadMediaAsset.find(params[:upload_media_asset_id])
media_asset = upload_media_asset.media_asset
upload = upload_media_asset.upload
# XXX depends on CurrentUser
commentary = ArtistCommentary.new(
original_title: params[:artist_commentary_title],
original_description: params[:artist_commentary_desc],
translated_title: params[:translated_commentary_title],
translated_description: params[:translated_commentary_desc],
)
post = Post.new(
uploader: upload.uploader,
uploader_ip_addr: upload.uploader_ip_addr,
md5: media_asset.md5,
file_ext: media_asset.file_ext,
file_size: media_asset.file_size,
image_width: media_asset.image_width,
image_height: media_asset.image_height,
source: Sources::Strategies.find(upload.source, upload.referer_url).canonical_url || upload.source,
tag_string: params[:tag_string],
rating: params[:rating],
parent_id: params[:parent_id],
is_pending: !upload.uploader.can_upload_free? || params[:is_pending].to_s.truthy?,
artist_commentary: (commentary if commentary.any_field_present?),
)
end
module FileMethods
extend ActiveSupport::Concern

188
app/models/upload.rb
View File

@@ -1,100 +1,28 @@
# frozen_string_literal: true
class Upload < ApplicationRecord
class Error < StandardError; end
MAX_VIDEO_DURATION = 140
class FileValidator < ActiveModel::Validator
def validate(record)
validate_file_ext(record)
validate_integrity(record)
validate_md5_uniqueness(record)
validate_video_duration(record)
validate_resolution(record)
end
attr_accessor :file
def validate_file_ext(record)
if record.file_ext.in?(["bin", "swf"])
record.errors.add(:file_ext, "is invalid (only JPEG, PNG, GIF, MP4, and WebM files are allowed")
end
end
belongs_to :uploader, class_name: "User"
has_many :upload_media_assets, dependent: :destroy
has_many :media_assets, through: :upload_media_assets
def validate_integrity(record)
if record.file.is_corrupt?
record.errors.add(:file, "is corrupted")
end
end
validates :source, format: { with: %r{\Ahttps?://}i, message: "is not a valid URL" }, if: -> { source.present? }
validates :referer_url, format: { with: %r{\Ahttps?://}i, message: "is not a valid URL" }, if: -> { referer_url.present? }
def validate_md5_uniqueness(record)
if record.md5.nil?
return
end
md5_post = Post.find_by_md5(record.md5)
if md5_post.nil?
return
end
if record.replaced_post && record.replaced_post == md5_post
return
end
record.errors.add(:md5, "duplicate: #{md5_post.id}")
end
def validate_resolution(record)
resolution = record.image_width.to_i * record.image_height.to_i
if resolution > Danbooru.config.max_image_resolution
record.errors.add(:base, "image resolution is too large (resolution: #{(resolution / 1_000_000.0).round(1)} megapixels (#{record.image_width}x#{record.image_height}); max: #{Danbooru.config.max_image_resolution / 1_000_000} megapixels)")
elsif record.image_width > Danbooru.config.max_image_width
record.errors.add(:image_width, "is too large (width: #{record.image_width}; max width: #{Danbooru.config.max_image_width})")
elsif record.image_height > Danbooru.config.max_image_height
record.errors.add(:image_height, "is too large (height: #{record.image_height}; max height: #{Danbooru.config.max_image_height})")
end
end
def validate_video_duration(record)
if !record.uploader.is_admin? && record.file.is_video? && record.file.duration.to_i > MAX_VIDEO_DURATION
record.errors.add(:base, "video must not be longer than #{MAX_VIDEO_DURATION.seconds.inspect}")
end
end
end
attr_accessor :as_pending, :replaced_post, :file
belongs_to :uploader, :class_name => "User"
belongs_to :post, optional: true
has_one :media_asset, foreign_key: :md5, primary_key: :md5
before_validation :initialize_attributes, on: :create
before_validation :assign_rating_from_tags
# validates :source, format: { with: /\Ahttps?/ }, if: ->(record) {record.file.blank?}, on: :create
validates :rating, inclusion: { in: %w[q e s] }, allow_nil: true
validates :md5, confirmation: true, if: ->(rec) { rec.md5_confirmation.present? }
validates_with FileValidator, on: :file
serialize :context, JSON
after_create :async_process_upload!
scope :pending, -> { where(status: "pending") }
scope :preprocessed, -> { where(status: "preprocessed") }
scope :completed, -> { where(status: "completed") }
scope :uploaded_by, ->(user_id) { where(uploader_id: user_id) }
def initialize_attributes
self.uploader_id = CurrentUser.id
self.uploader_ip_addr = CurrentUser.ip_addr
self.server = Socket.gethostname
end
def self.visible(user)
if user.is_admin?
all
elsif user.is_anonymous?
completed
else
completed.or(where(uploader: user))
where(uploader: user)
end
end
@@ -111,98 +39,44 @@ class Upload < ApplicationRecord
status == "completed"
end
def is_preprocessed?
status == "preprocessed"
end
def is_preprocessing?
status == "preprocessing"
end
def is_duplicate?
status.match?(/duplicate: \d+/)
end
def is_errored?
status.match?(/error:/)
end
def sanitized_status
if is_errored?
status.sub(/DETAIL:.+/m, "...")
else
status
end
end
def duplicate_post_id
@duplicate_post_id ||= status[/duplicate: (\d+)/, 1]
end
end
concerning :SourceMethods do
def source=(source)
source = source.unicode_normalize(:nfc)
# percent encode unicode characters in urls
if source =~ %r{\Ahttps?://}i
source = Addressable::URI.normalized_encode(source) rescue source
end
super(source)
end
def source_url
return nil unless source =~ %r{\Ahttps?://}i
Addressable::URI.heuristic_parse(source) rescue nil
end
end
def self.search(params)
q = search_attributes(params, :id, :created_at, :updated_at, :source, :rating, :parent_id, :server, :md5, :server, :file_ext, :file_size, :image_width, :image_height, :referer_url, :uploader, :post)
if params[:source_matches].present?
q = q.where_like(:source, params[:source_matches])
end
if params[:has_post].to_s.truthy?
q = q.where.not(post_id: nil)
elsif params[:has_post].to_s.falsy?
q = q.where(post_id: nil)
end
if params[:status].present?
q = q.where_like(:status, params[:status])
end
if params[:backtrace].present?
q = q.where_like(:backtrace, params[:backtrace])
end
if params[:tag_string].present?
q = q.where_like(:tag_string, params[:tag_string])
end
q = search_attributes(params, :id, :created_at, :updated_at, :source, :referer_url, :uploader, :status, :backtrace, :upload_media_assets, :media_assets)
q.apply_default_order(params)
end
def assign_rating_from_tags
rating = PostQueryBuilder.new(tag_string).find_metatag(:rating)
if rating.present?
self.rating = rating.downcase.first
def async_process_upload!
if file.present?
ProcessUploadJob.perform_now(self)
else
ProcessUploadJob.perform_later(self)
end
end
def upload_as_pending?
as_pending.to_s.truthy?
end
def process_upload!
update!(status: "processing")
def has_commentary?
artist_commentary_title.present? || artist_commentary_desc.present? || translated_commentary_title.present? || translated_commentary_desc.present?
if file.present?
media_file = MediaFile.open(file.tempfile)
elsif source.present?
strategy = Sources::Strategies.find(source, referer_url)
media_file = strategy.download_file!(strategy.image_url)
else
raise "No file or source provided"
end
media_asset = MediaAsset.upload!(media_file)
update!(media_assets: [media_asset], status: "completed")
rescue Exception => e
update!(status: "error: #{e.message}", backtrace: e.backtrace.join("\n"))
raise
end
def self.available_includes
[:uploader, :post]
[:uploader, :upload_media_assets, :media_assets]
end
end

17
app/policies/post_policy.rb
View File

@@ -13,6 +13,10 @@ class PostPolicy < ApplicationPolicy
unbanned? && record.visible?
end
def create?
unbanned? && record.uploader == user
end
def revert?
update?
end
@@ -66,11 +70,14 @@ class PostPolicy < ApplicationPolicy
user.is_gold?
end
def permitted_attributes
[
:tag_string, :old_tag_string, :parent_id, :old_parent_id,
:source, :old_source, :rating, :old_rating, :has_embedded_notes,
].compact
def permitted_attributes_for_create
%i[upload_media_asset_id tag_string rating parent_id source is_pending
artist_commentary_desc artist_commentary_title translated_commentary_desc
translated_commentary_title]
end
def permitted_attributes_for_update
%i[tag_string old_tag_string parent_id old_parent_id source old_source rating old_rating has_embedded_notes]
end
def api_attributes

20
app/policies/upload_policy.rb
View File

@@ -1,8 +1,12 @@
# frozen_string_literal: true
class UploadPolicy < ApplicationPolicy
def create?
unbanned?
end
def show?
record.is_completed? || user.is_admin? || record.uploader_id == user.id
user.is_admin? || record.uploader_id == user.id
end
def batch?
@@ -13,23 +17,9 @@ class UploadPolicy < ApplicationPolicy
unbanned?
end
def preprocess?
unbanned?
end
def can_view_tags?
user.is_admin? || record.uploader_id == user.id
end
def permitted_attributes
%i[file source tag_string rating status parent_id artist_commentary_title
artist_commentary_desc referer_url
md5_confirmation as_pending translated_commentary_title translated_commentary_desc]
end
def api_attributes
attributes = super
attributes -= [:tag_string] unless can_view_tags?
attributes
end
end

6
app/views/sources/show.js.erb
View File

@@ -5,8 +5,8 @@ $(document).trigger("danbooru:update-source-data", {
related_tags_html: "<%= j render "related_tags/source_tags", source: @source %>",
});
if ($("#c-uploads #a-new").length) {
$("#upload_artist_commentary_title").val(<%= raw @source.dtext_artist_commentary_title.to_json %>);
$("#upload_artist_commentary_desc").val(<%= raw @source.dtext_artist_commentary_desc.to_json %>);
if ($("#c-uploads #a-show").length) {
$("#post_artist_commentary_title").val(<%= raw @source.dtext_artist_commentary_title.to_json %>);
$("#post_artist_commentary_desc").val(<%= raw @source.dtext_artist_commentary_desc.to_json %>);
Danbooru.Upload.toggle_commentary();
}

15
app/views/uploads/_dropzone_preview.html.erb
View File

@@ -1,15 +0,0 @@
<div class="dz-preview dz-file-preview">
<img data-dz-thumbnail>
<div class="dz-details">
<div class="dz-filename">
<span data-dz-name></span>
</div>
<div class="dz-size" data-dz-size></div>
</div>
<div class="dz-progress">
<span class="dz-upload" data-dz-uploadprogress></span>
</div>
<div class="dz-error-message">
<span data-dz-errormessage></span>
</div>
</div>

22
app/views/uploads/_image.html.erb
View File

@@ -1,22 +0,0 @@
<% if params[:url] %>
<p id="upload-image-metadata">
<strong>Size</strong>
<% if @remote_size.present? %>
<span id="upload-image-metadata-filesize"><%= number_to_human_size(@remote_size) %></span>
<% end %>
<span id="upload-image-metadata-resolution"></span>
<span id="upload-image-metadata-size-links">
(<a id="upload-image-view-small" href="">small</a> | <a id="upload-image-view-large" href="">large</a> | <a id="upload-image-view-full" href="">full</a>)
</span>
</p>
<div id="upload-image">
<% if ImageProxy.needs_proxy?(@source.image_url) %>
<%= tag.img src: image_proxy_uploads_path(url: @source.image_url), title: "Preview", id: "image", class: "fit-width fit-height", onerror: "Danbooru.Upload.no_image_available()", "data-shortcut": "z" %>
<% elsif @source.image_url.present? %>
<%= tag.img src: @source.image_url, title: "Preview", id: "image", referrerpolicy: "no-referrer", class: "fit-width fit-height", onerror: "Danbooru.Upload.no_image_available()", "data-shortcut": "z" %>
<% end %>
<div id="no-image-available">No image preview available</div>
</div>
<% end %>

6
app/views/uploads/_secondary_links.html.erb
View File

@@ -1,7 +1,7 @@
<% content_for(:secondary_links) do %>
<%= subnav_link_to "Listing", uploads_path %>
<%= subnav_link_to "New", new_upload_path %>
<%= subnav_link_to "My Uploads", uploads_path %>
<%= subnav_link_to "New Upload", new_upload_path %>
<%= subnav_link_to "Batch Upload", batch_uploads_path %>
<%= subnav_link_to "Similar Images Search", iqdb_queries_path %>
<%= subnav_link_to "Reverse Image Search", iqdb_queries_path %>
<%= subnav_link_to "Help", wiki_page_path("help:upload") %>
<% end %>

91
app/views/uploads/index.html.erb
View File

@@ -1,61 +1,70 @@
<div id="c-uploads">
<div id="a-index">
<%= render "uploads/search" %>
<%= render "posts/partials/common/inline_blacklist" %>
<%= search_form_for(uploads_path) do |f| %>
<%= f.input :uploader_name, label: "Uploader", input_html: { value: params[:search][:uploader_name], data: { autocomplete: "user" } } %>
<%= f.input :source_like, label: "Source", input_html: { value: params[:search][:source_like] } %>
<%= f.input :status_like, label: "Status", collection: [%w[Completed completed], %w[Processing processing], %w[Pending pending], %w[Error error*]], include_blank: true, selected: params[:search][:status_like] %>
<%= f.submit "Search" %>
<% end %>
<%= table_for @uploads, class: "striped autofit", width: "100%" do |t| %>
<% t.column "Upload" do |upload| %>
<%= post_preview(upload.post, tags: "user:#{upload.uploader.name}", show_deleted: true) %>
<% end %>
<% t.column "Info", td: {class: "col-expand upload-info"} do |upload| %>
<span class="info">
<strong>Upload</strong>
<%= link_to "##{upload.id}", upload %>
</span>
<span class="info">
<strong>Rating</strong>
<%= upload.rating %>
</span>
<% if upload.post.present? %>
<span class="info">
<strong>Size</strong>
<%= link_to "#{upload.post.file_size.to_formatted_s(:human_size, precision: 4)} #{upload.post.file_ext}", upload.post.file_url %>
(<%= upload.post.image_width %>x<%= upload.post.image_height %>)
</span>
<% t.column "File" do |upload| %>
<% upload.media_assets.first.tap do |media_asset| %>
<% if media_asset.present? %>
<%= link_to upload, class: "inline-block" do %>
<%= tag.img src: media_asset.variant("180x180").file_url %>
<% end %>
<% end %>
<% end %>
<br>
<% end %>
<span class="info">
<% t.column "Info", td: {class: "col-expand upload-info"} do |upload| %>
<div>
<strong>Upload</strong>
<span><%= link_to "##{upload.id}", upload %></span>
</div>
<div>
<strong>Source</strong>
<%= link_to_if (upload.source =~ %r!\Ahttps?://!i), (upload.source.presence.try(:truncate, 50) || content_tag(:em, "none")), upload.source %>
<%= link_to "»", uploads_path(search: params[:search].merge(source_matches: upload.source)) %>
</span>
<br>
<span>
<% if upload.source.present? %>
<%= external_link_to upload.source %>
<%= link_to "»", uploads_path(search: params[:search].merge(source_like: upload.source)) %>
<% else %>
<em>none</em>
<% end %>
</span>
</div>
<% if upload.referer_url.present? %>
<span class="info">
<strong>Referer</strong>
<%= URI.parse(upload.referer_url).host rescue nil %>
</span>
<br>
<div>
<strong>Referrer</strong>
<span>
<%= external_link_to upload.referer_url %>
<%= link_to "»", uploads_path(search: params[:search].merge(referer_url: upload.referer_url)) %>
</span>
</div>
<% end %>
<% if policy(upload).can_view_tags? %>
<span class="info">
<strong>Tags</strong>
<%= render_inline_tag_list_from_names(upload.tag_string.split) %>
</span>
<% if upload.is_errored? %>
<div>
<strong>Error</strong>
<span>
<%= upload.status.delete_prefix("error: ") %>
</span>
</div>
<% end %>
<% end %>
<% t.column "Uploader" do |upload| %>
<%= link_to_user upload.uploader %>
<%= link_to "»", uploads_path(search: params[:search].merge(uploader_name: upload.uploader.name)) %>
<br><%= time_ago_in_words_tagged upload.created_at %>
<div><%= time_ago_in_words_tagged upload.created_at %></div>
<% end %>
<% t.column "Status", td: {class: "col-normal"} do |upload| %>
<%= render_status(upload) %>
<% t.column :status do |upload| %>
<%= upload.is_errored? ? "error" : upload.status %>
<% end %>
<% end %>

97
app/views/uploads/new.html.erb
View File

@@ -1,100 +1,11 @@
<%= content_for :html_header do %>
<meta name="max-file-size" content="<%= Danbooru.config.max_file_size %>">
<% end %>
<div id="c-uploads">
<div id="a-new" data-image-size="small">
<h1>Upload</h1>
<h1 class="text-center mb-4">Upload</h1>
<% if CurrentUser.user.upload_limit.limited? %>
<h2 style="margin-bottom: 1em;">You have reached your upload limit</h2>
<% else %>
<%= embed_wiki("help:upload_notice", id: "upload-guide-notice") %>
<% unless CurrentUser.can_upload_free? %>
<p id="upload-limit">
Upload Limit: <%= render "users/upload_limit", user: CurrentUser.user %>
</p>
<% end %>
<%= render "image" %>
<%= render "related_posts", source: @source %>
<%= render_source_data(nil) %>
<div id="client-errors" class="error-messages ui-state-error ui-corner-all" style="display:none"></div>
<%= edit_form_for(@upload, html: { id: "form" }) do |f| %>
<%= f.input :md5_confirmation, as: :hidden %>
<%= f.input :referer_url, as: :hidden, input_html: { value: params[:ref] } %>
<% if CurrentUser.can_upload_free? %>
<%= f.input :as_pending, as: :boolean, label: "Upload for approval", input_html: { checked: params[:as_pending].present? } %>
<% end %>
<%= f.input :file, as: :file, size: 50, wrapper_html: { class: "fallback" } %>
<div class="input" id="filedropzone">
<div class="dropzone-hint">Click or drag and drop to upload a file</div>
<div class="dropzone-hint hint">Max size: <%= number_to_human_size(Danbooru.config.max_file_size) %>. Supported filetypes: jpg, png, gif, mp4, webm.</div>
</div>
<div class="input string optional upload_source">
<%= f.label :source %>
<%= f.input_field :source, as: :string, placeholder: "Enter the URL to upload here", value: params[:url] %>
<%= tag.button "Similar", id: "similar-button" %>
</div>
<%= f.input :rating, collection: [["Explicit", "e"], ["Questionable", "q"], ["Safe", "s"]], as: :radio_buttons, selected: params[:rating] %>
<%= f.input :parent_id, label: "Parent ID", as: :string, input_html: { value: params[:parent_id] } %>
<div class="input upload_artist_commentary_container">
<strong>Commentary</strong>
<a href="#" id="toggle-artist-commentary">show »</a>
<div class="artist-commentary" style="display: none;">
<%= f.input :artist_commentary_title, as: :string, label: "Original Title", input_html: { value: params[:artist_commentary_title] } %>
<%= f.input :artist_commentary_desc, as: :text, label: "Original Description", input_html: { value: params[:artist_commentary_desc] } %>
</div>
</div>
<div class="input upload_commentary_translation_container" style="display: none;">
<strong>Translation</strong>
<a href="#" id="toggle-commentary-translation">show »</a>
<div class="commentary-translation" style="display: none;">
<%= f.input :translated_commentary_title, as: :string, label: "Translated Title", input_html: { value: params[:translated_commentary_title] } %>
<%= f.input :translated_commentary_desc, as: :text, label: "Translated Description", input_html: { value: params[:translated_commentary_desc] } %>
</div>
</div>
<div class="input fixed-width-container" id="tags-container">
<div class="header">
<%= f.label :tag_string, "Tags" %>
<span data-tag-counter data-for="#upload_tag_string">
<span class="tag-count"></span>
<img>
</span>
<a href="javascript:void(0)">
<%= external_link_icon(id: "open-edit-dialog", "data-shortcut": "shift+e") %>
</a>
</div>
<%= f.input :tag_string, label: false, hint: "Ctrl+Enter to submit", input_html: { "data-autocomplete": "tag-edit", "data-shortcut": "e", value: params[:tag_string] } %>
<%= render "related_tags/buttons" %>
</div>
<%= f.submit "Submit", id: "submit-button", data: { disable_with: false } %>
<%= render "related_tags/container" %>
<% end %>
<% end %>
<div class="flex items-center justify-center">
<%= render FileUploadComponent.new(url: @upload.source, referer_url: @upload.referer_url, drop_target: "body") %>
</div>
</div>
</div>
<template id="dropzone-preview-template">
<%= render "dropzone_preview" %>
</template>
<%= render "uploads/secondary_links" %>

143
app/views/uploads/show.html.erb
View File

@@ -1,51 +1,110 @@
<div id="c-uploads">
<div id="a-show">
<h1>Upload #<%= @upload.id %></h1>
<div id="a-show" data-image-size="small">
<h1>Upload</h1>
<ul>
<li>Date: <%= @upload.created_at %></li>
<li>Source: <%= @upload.source %></li>
<% if policy(@upload).can_view_tags? %>
<li>Tags: <%= @upload.tag_string %></li>
<% end %>
<% if @upload.md5.present? %>
<li>MD5: <%= @upload.md5 %>
<% end %>
<% if @upload.file_size.present? %>
<li>
Size: <%= number_to_human_size(@upload.file_size) %></li>
<% if @upload.image_width.present? %>
(<%= @upload.image_width %>x<%= @upload.image_height %>)
<% end %>
</li>
<% end %>
</ul>
<% if @upload.is_completed? %>
<p>This upload has finished processing. <%= link_to "View the post", post_path(@upload.post_id) %>.</p>
<% elsif @upload.is_pending? %>
<p>This upload is waiting to be processed. Please wait a few seconds.</p>
<% elsif @upload.is_processing? || @upload.is_preprocessing? || @upload.is_preprocessed? %>
<p>This upload is being processed. Please wait a few seconds.</p>
<% elsif @upload.is_duplicate? %>
<p>This upload is a duplicate: <%= link_to "post ##{@upload.duplicate_post_id}", post_path(@upload.duplicate_post_id) %></p>
<% else %>
<p>An error occurred: <%= render_status(@upload) %></p>
<% if CurrentUser.user.is_builder? %>
<%= render "static/backtrace", backtrace: @upload.backtrace.to_s.split(/\n/) %>
<% if @upload.is_pending? || @upload.is_processing? %>
<% content_for(:html_header) do %>
<meta http-equiv="refresh" content="1">
<% end %>
<% end %>
<p>
You can <%= link_to "upload another file", new_upload_path %> or <%= link_to "view your current uploads", uploads_path(:search => {:uploader_id => CurrentUser.id}) %>.
</p>
<% if @upload.is_errored? %>
<p><%= @upload.status %></p>
<% elsif @upload.is_pending? && @upload.source.present? %>
<p>Preparing to upload <%= external_link_to @upload.source %>...</p>
<% elsif @upload.is_processing? && @upload.source.present? %>
<p>Processing <%= external_link_to @upload.source %>...</p>
<% elsif !@upload.is_completed? %>
<p>Processing upload...</p>
<% elsif CurrentUser.user.upload_limit.limited? %>
<p>You have reached your upload limit. Please wait for your pending uploads to be approved before uploading more.</p>
<p id="upload-limit">Upload Limit: <%= render "users/upload_limit", user: CurrentUser.user %></p>
<% else %>
<%= embed_wiki("help:upload_notice", id: "upload-guide-notice") %>
<% unless CurrentUser.can_upload_free? %>
<p id="upload-limit">Upload Limit: <%= render "users/upload_limit", user: CurrentUser.user %></p>
<% end %>
<div id="client-errors" class="error-messages ui-state-error ui-corner-all" style="display:none"></div>
<% @upload_media_asset = @upload.upload_media_assets.first %>
<% @media_asset = @upload_media_asset.media_asset %>
<p id="upload-image-metadata">
<strong>Size</strong>
<span id="upload-image-metadata-filesize"><%= number_to_human_size(@media_asset.file_size) %></span>
<span id="upload-image-metadata-resolution"><%= @media_asset.image_width %>x<%= @media_asset.image_height %></span>
<span id="upload-image-metadata-size-links">
(<a id="upload-image-view-small" href="">small</a> | <a id="upload-image-view-large" href="">large</a> | <a id="upload-image-view-full" href="">full</a>)
</span>
</p>
<div id="upload-image">
<%= tag.img src: @media_asset.variant("original").file_url, title: "Preview", id: "image", class: "fit-width fit-height", "data-shortcut": "z" %>
</div>
<% if @upload.source.present? %>
<% @source = ::Sources::Strategies.find(@upload.source, @upload.referer_url) %>
<%= render "uploads/related_posts", source: @source %>
<%= render_source_data(@source) %>
<% end %>
<%= edit_form_for(@post, html: { id: "form" }) do |f| %>
<%= f.input :upload_media_asset_id, as: :hidden, input_html: { value: @upload_media_asset.id } %>
<%= f.input :source, as: :string %>
<%= f.input :rating, collection: [["Explicit", "e"], ["Questionable", "q"], ["Safe", "s"]], as: :radio_buttons, selected: @post.rating %>
<%= f.input :parent_id, label: "Parent ID", as: :string, input_html: { value: @post.parent_id } %>
<div class="input upload_artist_commentary_container">
<strong>Commentary</strong>
<a href="#" id="toggle-artist-commentary">show »</a>
<div class="artist-commentary" style="display: none;">
<%= f.input :artist_commentary_title, as: :string, label: "Original Title", input_html: { value: @post&.artist_commentary&.original_title } %>
<%= f.input :artist_commentary_desc, as: :text, label: "Original Description", input_html: { value: @post&.artist_commentary&.original_description } %>
</div>
</div>
<div class="input upload_commentary_translation_container" style="display: none;">
<strong>Translation</strong>
<a href="#" id="toggle-commentary-translation">show »</a>
<div class="commentary-translation" style="display: none;">
<%= f.input :translated_commentary_title, as: :string, label: "Translated Title", input_html: { value: @post&.artist_commentary&.translated_title } %>
<%= f.input :translated_commentary_desc, as: :text, label: "Translated Description", input_html: { value: @post&.artist_commentary&.translated_description } %>
</div>
</div>
<div class="input fixed-width-container" id="tags-container">
<div class="header">
<%= f.label :tag_string, "Tags" %>
<span data-tag-counter data-for="#post_tag_string">
<span class="tag-count"></span>
<img>
</span>
<a href="javascript:void(0)">
<%= external_link_icon(id: "open-edit-dialog", "data-shortcut": "shift+e") %>
</a>
</div>
<%= f.input :tag_string, label: false, hint: "Ctrl+Enter to submit", input_html: { "data-autocomplete": "tag-edit", "data-shortcut": "e", value: @post.tag_string } %>
<%= render "related_tags/buttons" %>
</div>
<%= f.submit "Upload", id: "submit-button", data: { disable_with: false } %>
<% if CurrentUser.can_upload_free? %>
<%= f.input :is_pending, as: :boolean, label: "Upload for approval", wrapper_html: { class: "inline-block" }, input_html: { checked: @post.is_pending? } %>
<% end %>
<%= render "related_tags/container" %>
<% end %>
<% end %>
</div>
</div>
<%= render "uploads/secondary_links" %>
<% if @upload.is_pending? || @upload.is_processing? || @upload.is_preprocessing? || @upload.is_preprocessed? %>
<% content_for(:html_header) do %>
<meta http-equiv="refresh" content="2">
<% end %>
<% end %>

1
app/views/uploads/update.js.erb
View File

@@ -1 +0,0 @@
location.reload();