* Save the filename for files uploaded from disk. This could be used in
the future to extract source data if the filename is from a known site.
* Save both the image URL and the page URL for files uploaded from
source. This is needed for multi-file uploads. The image URL is the
URL of the file actually downloaded from the source. This can be
different from the URL given by the user, if the user tried to upload
a sample URL and we automatically changed it to the original URL. The
page URL is the URL of the page containing the image. We don't always
know this, for example if someone uploads a Twitter image without the
bookmarklet, then we can't find the page URL.
* Add a fix script to backfill URLs for existing uploads. For file
uploads, the filename will be set to "unknown.jpg". For source
uploads, we fetch the source data again to get the image and page
URLs. This may fail for uploads that have been deleted from the
source since uploading.
This exception was thrown by app/logical/pixiv_ajax_client.rb:406 when a
Pixiv API call failed with a network error. In this case we tried to log
the response body, but this failed because we returned a faked HTTP
response with an empty string for the body, which the http.rb library
didn't like because it was expecting an IO-like object for the body.
On the post index page, show the "Artist" tab instead of the "Wiki" tab when searching for
an artist tag that doesn't have an artist entry. This way the user is prompted to create a
new artist entry instead of a new wiki.
Fix URLs being normalized after checking for duplicates rather than
before, which meant that URLs that differed in capitalization weren't
detected as duplicates.
Fix requests for non-existent .js pages, for example https://danbooru.donmai.us/oaisfj.js,
raising AbstractController::DoubleRenderError when trying to render the 404 response.
Fix a potential exploit where private information could be leaked if
it was contained in the error message of an unexpected exception.
For example, NoMethodError contains a raw dump of the object in the
error message, which could leak private user data if you could force a
User object to raise a NoMethodError.
Fix the error page to only show known-safe error messages from expected
exceptions, not unknown error messages from unexpected exceptions.
API changes:
* JSON errors now have a `message` param. The message will be blank for unknown exceptions.
* XML errors have a new format. This is a breaking change. They now look like this:
<result>
<success type="boolean">false</success>
<error>PaginationExtension::PaginationError</error>
<message>You cannot go beyond page 5000.</message>
<backtrace type="array">
<backtrace>app/logical/pagination_extension.rb:54:in `paginate'</backtrace>
<backtrace>app/models/application_record.rb:17:in `paginate'</backtrace>
<backtrace>app/logical/post_query_builder.rb:529:in `paginated_posts'</backtrace>
<backtrace>app/logical/post_sets/post.rb:95:in `posts'</backtrace>
<backtrace>app/controllers/posts_controller.rb:22:in `index'</backtrace>
</backtrace>
</result>
instead of like this:
<result success="false">You cannot go beyond page 5000.</result>
`string.mb_chars.downcase` was used to correctly downcase Unicode
characters when downcasing strings in Ruby <2.4. This hasn't been needed
since Ruby 2.4.
Mark old columns as ignored in preparation for dropping them. Make the
rating and tag_string nullable so they don't have to be set when
creating uploads and can be ignored too.
Lock the post during replacement to ensure we have the latest version of
the tags and to ensure nobody else can modify the post until after the
replacement is finished.
Perform the replacement in a before_create callback so that it runs in a
transaction and if it fails, the transaction will rollback and the
replacement record won't be created.
Doing the replacement in a transaction isn't great because, for one
thing, it could hold the transaction open a long time, which isn't good
for the database. And two, if the transaction rolls back, the database
changes will be undone, but if the replacement file has already been saved
to disk, then it won't be undone, which could result in a dangling file.
* Fix a bug where creating posts failed if IQDB wasn't configured.
* Fix broken Skeb test caused by changed URL.
* Fix broken IP geolocation tests caused by API returning different data.
* Fix broken post regeneration tests.
* Move replacement tests from test/unit/upload_service_test.rb to
test/functional/post_replacement_controller_test.rb
* Move UploadService::Replacer to PostReplacementProcessor.
* Fix a minor bug where if you used the API to replace a post with a file,
the replacement would fail unless you passed an empty string for the
replacement_url.
Fix strings like "pokémon" (NFD form) and "pokémon" (NFC form) being
considered different strings in sources.
Also add a fix script to fix existing sources. There were only 15 posts
with unnormalized sources.
* On the upload page, show the video when uploading a video or ugoira.
* On the upload page, show the filesize and resolution beneath the
image, instead of above it.
* On the media asset show page, show the full video or ugoira instead of
just the thumbnail.
Add a view component for rendering thumbnails for media assets.
This lets us properly show thumbnails on the upload listing page and the
media assets listing page, including support for high pixel density thumbnails
and video length icons for videos.
Fixes not being able to see thumbnails on the /media_assets page.
This is mostly copy/pasted from the post preview component. FIXME: don't duplicate code.
Automatically merge tags when uploading a duplicate.
There are two cases:
* You try to upload an image, but it's already on Danbooru. In this case
you'll be immediately redirected to the original post, before you
can start tagging the upload.
* You're uploading an image, it wasn't a dupe when you first opened the
upload page, but you got sniped while tagging it. In this case your tags
will be merged with the original post, and you will be redirected to the
original post.
There are a few corner cases:
* If you don't have permission to edit the original post, for example
because it's banned or has a censored tag, then your tags won't be
merged and will be silently ignored.
* Only the tags, rating, and parent ID will be merged. The source and
artist commentary won't be merged. This is so that if an artist uploads
the exact same file to multiple sites, the new source won't override
the original source.
* Some tags might be contradictory. For example, the new post might
be tagged translation_request, but the original post might already be
translated. It's up to the user to fix these things afterwards.
Fix not being able to change the post's source when submitting the
upload. For example, if you were uploading a Twitter image from a direct
Twitter image URL, and you tried to change the source to the tweet URL
on the upload page before creating the post, then the source would be
ignored when the post was created.
* Fix broken upload tests.
* Fix uploads to return an error if both a file and a source are given
at the same time, or if neither are given. Also fix the error message
in this case so that it doesn't include "base" at the start of the string.
* Fix uploads to percent-encode any Unicode characters in the source URL.
* Add a max filesize validation to media assets.
Rework the upload process so that files are saved to Danbooru first
before the user starts tagging the upload.
The main user-visible change is that you have to select the file first
before you can start tagging it. Saving the file first lets us fix a
number of problems:
* We can check for dupes before the user tags the upload.
* We can perform dupe checks and show preview images for users not using the bookmarklet.
* We can show preview images without having to proxy images through Danbooru.
* We can show previews of videos and ugoira files.
* We can reliably show the filesize and resolution of the image.
* We can let the user save files to upload later.
* We can get rid of a lot of spaghetti code related to preprocessing
uploads. This was the cause of most weird "md5 confirmation doesn't
match md5" errors.
(Not all of these are implemented yet.)
Internally, uploading is now a two-step process: first we create an upload
object, then we create a post from the upload. This is how it works:
* The user goes to /uploads/new and chooses a file or pastes an URL into
the file upload component.
* The file upload component calls `POST /uploads` to create an upload.
* `POST /uploads` immediately returns a new upload object in the `pending` state.
* Danbooru starts processing the upload in a background job (downloading,
resizing, and transferring the image to the image servers).
* The file upload component polls `/uploads/$id.json`, checking the
upload `status` until it returns `completed` or `error`.
* When the upload status is `completed`, the user is redirected to /uploads/$id.
* On the /uploads/$id page, the user can tag the upload and submit it.
* The upload form calls `POST /posts` to create a new post from the upload.
* The user is redirected to the new post.
This is the data model:
* An upload represents a set of files uploaded to Danbooru by a user.
Uploaded files don't have to belong to a post. An upload has an
uploader, a status (pending, processing, completed, or error), a
source (unless uploading from a file), and a list of media assets
(image or video files).
* There is a has-and-belongs-to-many relationship between uploads and
media assets. An upload can have many media assets, and a media asset
can belong to multiple uploads. Uploads are joined to media assets
through a upload_media_assets table.
An upload could potentially have multiple media assets if it's a Pixiv
or Twitter gallery. This is not yet implemented (at the moment all
uploads have one media asset).
A media asset can belong to multiple uploads if multiple people try
to upload the same file, or if the same user tries to upload the same
file more than once.
New features:
* On the upload page, you can press Ctrl+V to paste an URL and immediately upload it.
* You can save files for upload later. Your saved files are at /uploads.
Fixes:
* Improved error messages when uploading invalid files, bad URLs, and
when forgetting the rating.
Fix regression introduced in 0db20e0ca. Setting `format: false` on the
wiki pages resource disabled format negotiation on all wiki page routes,
not just the show page, which meant /wiki_pages.json no longer worked.
The fix to monkey patch the internal Rails method that parses the file
extension from the URL, and have it ignore everything but the .html,
.json, .js, and .xml extensions. This is really hacky and may break in
future Rails releases.
Fix it so that when a forum topic is deleted, all posts in the topic are
deleted too. Also make it so that when a forum topic is undeleted, all
posts in it are undeleted too.
Before when a topic was deleted, only the topic itself was marked as
deleted, not the posts inside the topic. This meant that when a spam
topic was deleted, the OP wouldn't be marked as deleted, so any
modreports against it wouldn't be marked as handled.
Also change it so that it's not possible to undelete a post in a deleted
topic, or to delete the OP of a topic without deleting the topic itself.
Finally, add a fix script to delete all active posts in deleted topics,
and to undelete all deleted OPs in active topics.
* Add ability to mark moderation reports as 'handled' or 'rejected'.
* Automatically mark reports as handled when the comment or forum post
is deleted.
* Send a dmail to the reporter when their report is handled.
* Don't show the report notice on comments or forum posts when all
reports against it have been handled or rejected.
* Add a fix script to mark all existing reports for deleted comments,
forum posts, or dmails as handled.
Normalize artist group names following the same rules as artist other names.
This means artist group names now use underscores instead of spaces.
It also means extra space characters at the beginning and end of names
is stripped, and Unicode characters are normalized.
Fixes#4647, which was caused by users accidentally replacing group
names with a single space character when trying to remove a group.
Fix wiki pages like this returning 406 errors:
* https://danbooru.donmai.us/wiki_pages/rnd.jpg
Caused by Rails parsing the .jpg part as a file extension and trying to
return a JPEG in response. This happens deep in Rails' MIME negotiation
code, so it's hard to override. The fix is to pass `format: false` in
the route to disable all special handling of file extensions by Rails,
and then handle it ourselves in the controller. Ugly.
This only affected two tags: `rnd.jpg` and `haru.jpg`.
Autotag non-web_source on posts that have a non-http:// or https:// URL.
Add a fix script to backfill old posts.
Syntactically invalid URLs are still considered web sources. For
example, `https://google,com` technically isn't a valid URL, but it's
not considered a non-web source.
Fix the /posts index controller not logging the normalized search query
to NewRelic when the search failed, either because of a tag limit error,
a search timeout, or a RSS feed rate limit error.
Also don't log the number of search results when it's an API request or
failed search. This is to avoid doing a potentially slow full post count
when it's not otherwise needed.
