Factor out the Stripe code from the UserUpgrade class. Introduce a new
PaymentTransaction abstract class that represents a payment with some
payment processor, and a PaymentTransaction::Stripe class that
implements transactions with Stripe.
Note that we can't completely eliminate Stripe even though we no longer
accept payments with it because we still need to be able to look up old
payments in Stripe.
Show a "This page has been removed because of a takedown request" error when
an unauthorized user searches for a banned tag, or tries to view a banned post.
Raise an error if the search is invalid for one of the following reasons:
* It contains multiple conflicting order: metatags (e.g. `order:score order:favcount` or `ordfav:a ordfav:b`).
* It contains a metatag that can't be used more than once: (e.g. `limit:5 limit:10`, `random:5 random:10`).
* It contains a metatag that can't be negated (e.g. `-order:score`, `-limit:20`, or `-random:20`).
* It contains a metatag that can't be used in an OR clause (e.g. ` touhou or order:score`, `touhou or limit:20`, `touhou or random:20`).
Show a warning when creating a duplicate artist; that is, when adding a
URL that already belongs to another artist.
This is a soft warning rather than a hard error because there are some
cases where multiple artists legitimately share the same site or account.
Forcibly redirect users to the name change page if their name is
invalid. This means user with invalid names can't do anything or view
any pages until they change their name. API requests are still allowed.
* Add a "Size" menu to the My Uploads / All Uploads pages to allow
changing the thumbnail size.
* Make the My Uploads / All Uploads pages use the same thumbnail size as
the post index page.
* Change the "Gallery | Table" links on the My Uploads page to icons.
Fix the "My Uploads" page showing Admins all uploads, not just their own
uploads.
Changes the URL of the My Uploads page from /uploads to /users/:id/uploads.
This page shows each individual file you've uploaded. This is different
from the regular uploads page because files in multi-file uploads are
not grouped together.
* Make thumbnails on the "My Uploads" page show an icon with an image
count when an upload contains multiple files.
* Make the "My Uploads" page show each upload, not each individual file.
If an upload contains multiple files, they're shown grouped together
under a single upload. This does mean that failed or duplicate uploads
will show up on this page now. This is because this page shows each
upload attempt, not each uniquely uploaded file.
Include media assets in /uploads.json and /uploads/:id.json API responses, like this:
{
"id": 4983629,
"source": "https://www.pixiv.net/en/artworks/96198438",
"uploader_id": 52664,
"status": "completed",
"created_at": "2022-02-12T16:26:04.680-06:00",
"updated_at": "2022-02-12T16:26:08.071-06:00",
"referer_url": "",
"error": null,
"media_asset_count": 1,
"upload_media_assets": [
{
"id": 9370,
"created_at": "2022-02-12T16:26:08.068-06:00",
"updated_at": "2022-02-12T16:26:08.068-06:00",
"upload_id": 4983629,
"media_asset_id": 5206552,
"status": "pending",
"source_url": "https://i.pximg.net/img-original/img/2022/02/13/01/20/19/96198438_p0.jpg",
"error": null,
"page_url": "https://www.pixiv.net/artworks/96198438",
"media_asset": {
"id": 5206552,
"created_at": "2022-02-12T16:26:07.980-06:00",
"updated_at": "2022-02-12T16:26:08.061-06:00",
"md5": "90a85a5fae5f0e86bdb2501229af05b7",
"file_ext": "jpg",
"file_size": 1055775,
"image_width": 1052,
"image_height": 1545,
"duration": null,
"status": "active"
}
}
]
}
This is needed so you can check for upload errors in the API, since in a multi-file
upload, each asset can have a separate error message. This is a stopgap solution until
something like /uploads.json?include=upload_media_assets.media_asset works.
Fix requests for non-existent .js pages, for example https://danbooru.donmai.us/oaisfj.js,
raising AbstractController::DoubleRenderError when trying to render the 404 response.
Fix two issues that could lead to duplicate errors when creating posts:
* Fix the submit button on the upload form to disable itself on submit, to prevent
accidental double submit errors.
* Fix a race condition when checking for MD5 duplicates. MD5 uniqueness is checked on both
the Rails level, with a uniqueness validation, and on the database level, with a unique
index on the md5 column. Creating a post could fail with an ActiveRecord::RecordNotUnique
error if the uniqueness validation in Rails passed, but the uniqueness constraint in the
database failed. In this case, we catch the RecordNotUnique error and convert it to a
Rails validation error so we can treat it like a normal validation failure.
Fix a potential exploit where private information could be leaked if
it was contained in the error message of an unexpected exception.
For example, NoMethodError contains a raw dump of the object in the
error message, which could leak private user data if you could force a
User object to raise a NoMethodError.
Fix the error page to only show known-safe error messages from expected
exceptions, not unknown error messages from unexpected exceptions.
API changes:
* JSON errors now have a `message` param. The message will be blank for unknown exceptions.
* XML errors have a new format. This is a breaking change. They now look like this:
<result>
<success type="boolean">false</success>
<error>PaginationExtension::PaginationError</error>
<message>You cannot go beyond page 5000.</message>
<backtrace type="array">
<backtrace>app/logical/pagination_extension.rb:54:in `paginate'</backtrace>
<backtrace>app/models/application_record.rb:17:in `paginate'</backtrace>
<backtrace>app/logical/post_query_builder.rb:529:in `paginated_posts'</backtrace>
<backtrace>app/logical/post_sets/post.rb:95:in `posts'</backtrace>
<backtrace>app/controllers/posts_controller.rb:22:in `index'</backtrace>
</backtrace>
</result>
instead of like this:
<result success="false">You cannot go beyond page 5000.</result>
Fix a bug where images on the My Uploads page would link to the wrong
upload. If an image had been uploaded by multiple users, then it would
link to the first upload belonging to the first person to upload the
image. This would lead to an Access Denied error when you tried to open
the upload.
Also fix a bug where uploads on the My Uploads page were ordered
incorrectly. They were ordered by most recent asset first, rather than
most recent upload first. This meant if you uploaded an image someone
else had already uploaded, then it would appear behind your other more
recent uploads.
Mark old columns as ignored in preparation for dropping them. Make the
rating and tag_string nullable so they don't have to be set when
creating uploads and can be ignored too.
Perform the replacement in a before_create callback so that it runs in a
transaction and if it fails, the transaction will rollback and the
replacement record won't be created.
Doing the replacement in a transaction isn't great because, for one
thing, it could hold the transaction open a long time, which isn't good
for the database. And two, if the transaction rolls back, the database
changes will be undone, but if the replacement file has already been saved
to disk, then it won't be undone, which could result in a dangling file.
* Move replacement tests from test/unit/upload_service_test.rb to
test/functional/post_replacement_controller_test.rb
* Move UploadService::Replacer to PostReplacementProcessor.
* Fix a minor bug where if you used the API to replace a post with a file,
the replacement would fail unless you passed an empty string for the
replacement_url.
Automatically merge tags when uploading a duplicate.
There are two cases:
* You try to upload an image, but it's already on Danbooru. In this case
you'll be immediately redirected to the original post, before you
can start tagging the upload.
* You're uploading an image, it wasn't a dupe when you first opened the
upload page, but you got sniped while tagging it. In this case your tags
will be merged with the original post, and you will be redirected to the
original post.
There are a few corner cases:
* If you don't have permission to edit the original post, for example
because it's banned or has a censored tag, then your tags won't be
merged and will be silently ignored.
* Only the tags, rating, and parent ID will be merged. The source and
artist commentary won't be merged. This is so that if an artist uploads
the exact same file to multiple sites, the new source won't override
the original source.
* Some tags might be contradictory. For example, the new post might
be tagged translation_request, but the original post might already be
translated. It's up to the user to fix these things afterwards.
Fix the upload page so that it shows similar images (IQDB matches) for
files uploaded from your computer. Before this only worked for files
uploaded from a source.
Rework the upload process so that files are saved to Danbooru first
before the user starts tagging the upload.
The main user-visible change is that you have to select the file first
before you can start tagging it. Saving the file first lets us fix a
number of problems:
* We can check for dupes before the user tags the upload.
* We can perform dupe checks and show preview images for users not using the bookmarklet.
* We can show preview images without having to proxy images through Danbooru.
* We can show previews of videos and ugoira files.
* We can reliably show the filesize and resolution of the image.
* We can let the user save files to upload later.
* We can get rid of a lot of spaghetti code related to preprocessing
uploads. This was the cause of most weird "md5 confirmation doesn't
match md5" errors.
(Not all of these are implemented yet.)
Internally, uploading is now a two-step process: first we create an upload
object, then we create a post from the upload. This is how it works:
* The user goes to /uploads/new and chooses a file or pastes an URL into
the file upload component.
* The file upload component calls `POST /uploads` to create an upload.
* `POST /uploads` immediately returns a new upload object in the `pending` state.
* Danbooru starts processing the upload in a background job (downloading,
resizing, and transferring the image to the image servers).
* The file upload component polls `/uploads/$id.json`, checking the
upload `status` until it returns `completed` or `error`.
* When the upload status is `completed`, the user is redirected to /uploads/$id.
* On the /uploads/$id page, the user can tag the upload and submit it.
* The upload form calls `POST /posts` to create a new post from the upload.
* The user is redirected to the new post.
This is the data model:
* An upload represents a set of files uploaded to Danbooru by a user.
Uploaded files don't have to belong to a post. An upload has an
uploader, a status (pending, processing, completed, or error), a
source (unless uploading from a file), and a list of media assets
(image or video files).
* There is a has-and-belongs-to-many relationship between uploads and
media assets. An upload can have many media assets, and a media asset
can belong to multiple uploads. Uploads are joined to media assets
through a upload_media_assets table.
An upload could potentially have multiple media assets if it's a Pixiv
or Twitter gallery. This is not yet implemented (at the moment all
uploads have one media asset).
A media asset can belong to multiple uploads if multiple people try
to upload the same file, or if the same user tries to upload the same
file more than once.
New features:
* On the upload page, you can press Ctrl+V to paste an URL and immediately upload it.
* You can save files for upload later. Your saved files are at /uploads.
Fixes:
* Improved error messages when uploading invalid files, bad URLs, and
when forgetting the rating.
