Fix two issues that could lead to duplicate errors when creating posts:
* Fix the submit button on the upload form to disable itself on submit, to prevent
accidental double submit errors.
* Fix a race condition when checking for MD5 duplicates. MD5 uniqueness is checked on both
the Rails level, with a uniqueness validation, and on the database level, with a unique
index on the md5 column. Creating a post could fail with an ActiveRecord::RecordNotUnique
error if the uniqueness validation in Rails passed, but the uniqueness constraint in the
database failed. In this case, we catch the RecordNotUnique error and convert it to a
Rails validation error so we can treat it like a normal validation failure.
Fix buttons appearing to be clickable when in the disabled state.
Submit buttons are normally disabled after a form is submitted. Before
these buttons would still look clickable. Now disabled buttons are greyed
out instead of looking the same as normal buttons.
Delete all old upload records from before the upload rework in abdab7a0a
/ f11c46b4f. Uploads from before the rework don't have any attached
media assets, so they're not valid under the new system because we can't
find which files they were for.
Before the rework, completed uploads were only saved for 1 hour, and
failed uploads were only saved for 3 days, so deleting this data
doesn't really lose anything that wouldn't have been deleted before.
Fix a potential exploit where private information could be leaked if
it was contained in the error message of an unexpected exception.
For example, NoMethodError contains a raw dump of the object in the
error message, which could leak private user data if you could force a
User object to raise a NoMethodError.
Fix the error page to only show known-safe error messages from expected
exceptions, not unknown error messages from unexpected exceptions.
API changes:
* JSON errors now have a `message` param. The message will be blank for unknown exceptions.
* XML errors have a new format. This is a breaking change. They now look like this:
<result>
<success type="boolean">false</success>
<error>PaginationExtension::PaginationError</error>
<message>You cannot go beyond page 5000.</message>
<backtrace type="array">
<backtrace>app/logical/pagination_extension.rb:54:in `paginate'</backtrace>
<backtrace>app/models/application_record.rb:17:in `paginate'</backtrace>
<backtrace>app/logical/post_query_builder.rb:529:in `paginated_posts'</backtrace>
<backtrace>app/logical/post_sets/post.rb:95:in `posts'</backtrace>
<backtrace>app/controllers/posts_controller.rb:22:in `index'</backtrace>
</backtrace>
</result>
instead of like this:
<result success="false">You cannot go beyond page 5000.</result>
Add more sensitive attributes to the filtered parameters list so that
they aren't shown in exception messages, and aren't logged in log files
or to NewRelic.
Only do this in production so that in testing and development, you can
still see these things when inspecting objects on the console.
Better fix for 8173c73aa. When not using the bookmarklet, make the back
button on the upload tagging page take you back to the new upload page.
When using the bookmarklet, make it take you back to the source.
`string.mb_chars.downcase` was used to correctly downcase Unicode
characters when downcasing strings in Ruby <2.4. This hasn't been needed
since Ruby 2.4.
Make uploads faster by generating and saving thumbnails in parallel.
We generate each thumbnail in parallel, then send each thumbnail to the
backend image servers in parallel.
Most images have 5 variants: 'preview' (150x150), 180x180, 360x360,
720x720, and 'sample' (850px width). Plus the original file, that's 6
files we have to save. In production we have 2 image servers, so we have
to save each file twice, to 2 remote servers. Doing all this in parallel
should make uploads significantly faster.
Fix bug in 2c0feaf94 that caused all the assets on the /uploads?mode=gallery
page to show the same source. The loop clobbered the `@upload` variable
such that the value from the last iteration was used by all assets.
Fix a bug where images on the My Uploads page would link to the wrong
upload. If an image had been uploaded by multiple users, then it would
link to the first upload belonging to the first person to upload the
image. This would lead to an Access Denied error when you tried to open
the upload.
Also fix a bug where uploads on the My Uploads page were ordered
incorrectly. They were ordered by most recent asset first, rather than
most recent upload first. This meant if you uploaded an image someone
else had already uploaded, then it would appear behind your other more
recent uploads.
Fix a bug where, if you used the bookmarklet to upload a file, and you
clicked on the back button on the upload tagging page, then you would be
taken back to the upload page, which would autosubmit the upload again.
Now if you click the back button on the upload tagging page, you'll be
taken back to the page where you used the bookmarklet, not the upload page.
Fix regression in 6fbca01a2.
Also make it so the "reply" link automatically inserts the cursor into
the reply box. Do this for both forum posts and comments. Before it only
did this when quoting a post, not when creating a new post.
Mark old columns as ignored in preparation for dropping them. Make the
rating and tag_string nullable so they don't have to be set when
creating uploads and can be ignored too.
Add a thumbnail view to the /media_assets page. This page lets you see
all images uploaded to Danbooru by all users (although you can't see who
the uploader is). Also add a link to this page in the subnav bar on the
upload page.
Lock the post during replacement to ensure we have the latest version of
the tags and to ensure nobody else can modify the post until after the
replacement is finished.
Perform the replacement in a before_create callback so that it runs in a
transaction and if it fails, the transaction will rollback and the
replacement record won't be created.
Doing the replacement in a transaction isn't great because, for one
thing, it could hold the transaction open a long time, which isn't good
for the database. And two, if the transaction rolls back, the database
changes will be undone, but if the replacement file has already been saved
to disk, then it won't be undone, which could result in a dangling file.
* Fix a bug where creating posts failed if IQDB wasn't configured.
* Fix broken Skeb test caused by changed URL.
* Fix broken IP geolocation tests caused by API returning different data.
* Fix broken post regeneration tests.
* Move replacement tests from test/unit/upload_service_test.rb to
test/functional/post_replacement_controller_test.rb
* Move UploadService::Replacer to PostReplacementProcessor.
* Fix a minor bug where if you used the API to replace a post with a file,
the replacement would fail unless you passed an empty string for the
replacement_url.
Fix strings like "pokémon" (NFD form) and "pokémon" (NFC form) being
considered different strings in sources.
Also add a fix script to fix existing sources. There were only 15 posts
with unnormalized sources.
* On the upload page, show the video when uploading a video or ugoira.
* On the upload page, show the filesize and resolution beneath the
image, instead of above it.
* On the media asset show page, show the full video or ugoira instead of
just the thumbnail.
Add a view component for rendering thumbnails for media assets.
This lets us properly show thumbnails on the upload listing page and the
media assets listing page, including support for high pixel density thumbnails
and video length icons for videos.
Fixes not being able to see thumbnails on the /media_assets page.
This is mostly copy/pasted from the post preview component. FIXME: don't duplicate code.
Automatically merge tags when uploading a duplicate.
There are two cases:
* You try to upload an image, but it's already on Danbooru. In this case
you'll be immediately redirected to the original post, before you
can start tagging the upload.
* You're uploading an image, it wasn't a dupe when you first opened the
upload page, but you got sniped while tagging it. In this case your tags
will be merged with the original post, and you will be redirected to the
original post.
There are a few corner cases:
* If you don't have permission to edit the original post, for example
because it's banned or has a censored tag, then your tags won't be
merged and will be silently ignored.
* Only the tags, rating, and parent ID will be merged. The source and
artist commentary won't be merged. This is so that if an artist uploads
the exact same file to multiple sites, the new source won't override
the original source.
* Some tags might be contradictory. For example, the new post might
be tagged translation_request, but the original post might already be
translated. It's up to the user to fix these things afterwards.
Fix not being able to change the post's source when submitting the
upload. For example, if you were uploading a Twitter image from a direct
Twitter image URL, and you tried to change the source to the tweet URL
on the upload page before creating the post, then the source would be
ignored when the post was created.
