Fix URLs being normalized after checking for duplicates rather than
before, which meant that URLs that differed in capitalization weren't
detected as duplicates.
`string.mb_chars.downcase` was used to correctly downcase Unicode
characters when downcasing strings in Ruby <2.4. This hasn't been needed
since Ruby 2.4.
* Fix a bug where creating posts failed if IQDB wasn't configured.
* Fix broken Skeb test caused by changed URL.
* Fix broken IP geolocation tests caused by API returning different data.
* Fix broken post regeneration tests.
* Move replacement tests from test/unit/upload_service_test.rb to
test/functional/post_replacement_controller_test.rb
* Move UploadService::Replacer to PostReplacementProcessor.
* Fix a minor bug where if you used the API to replace a post with a file,
the replacement would fail unless you passed an empty string for the
replacement_url.
Fix strings like "pokémon" (NFD form) and "pokémon" (NFC form) being
considered different strings in sources.
Also add a fix script to fix existing sources. There were only 15 posts
with unnormalized sources.
Automatically merge tags when uploading a duplicate.
There are two cases:
* You try to upload an image, but it's already on Danbooru. In this case
you'll be immediately redirected to the original post, before you
can start tagging the upload.
* You're uploading an image, it wasn't a dupe when you first opened the
upload page, but you got sniped while tagging it. In this case your tags
will be merged with the original post, and you will be redirected to the
original post.
There are a few corner cases:
* If you don't have permission to edit the original post, for example
because it's banned or has a censored tag, then your tags won't be
merged and will be silently ignored.
* Only the tags, rating, and parent ID will be merged. The source and
artist commentary won't be merged. This is so that if an artist uploads
the exact same file to multiple sites, the new source won't override
the original source.
* Some tags might be contradictory. For example, the new post might
be tagged translation_request, but the original post might already be
translated. It's up to the user to fix these things afterwards.
* Fix broken upload tests.
* Fix uploads to return an error if both a file and a source are given
at the same time, or if neither are given. Also fix the error message
in this case so that it doesn't include "base" at the start of the string.
* Fix uploads to percent-encode any Unicode characters in the source URL.
* Add a max filesize validation to media assets.
Rework the upload process so that files are saved to Danbooru first
before the user starts tagging the upload.
The main user-visible change is that you have to select the file first
before you can start tagging it. Saving the file first lets us fix a
number of problems:
* We can check for dupes before the user tags the upload.
* We can perform dupe checks and show preview images for users not using the bookmarklet.
* We can show preview images without having to proxy images through Danbooru.
* We can show previews of videos and ugoira files.
* We can reliably show the filesize and resolution of the image.
* We can let the user save files to upload later.
* We can get rid of a lot of spaghetti code related to preprocessing
uploads. This was the cause of most weird "md5 confirmation doesn't
match md5" errors.
(Not all of these are implemented yet.)
Internally, uploading is now a two-step process: first we create an upload
object, then we create a post from the upload. This is how it works:
* The user goes to /uploads/new and chooses a file or pastes an URL into
the file upload component.
* The file upload component calls `POST /uploads` to create an upload.
* `POST /uploads` immediately returns a new upload object in the `pending` state.
* Danbooru starts processing the upload in a background job (downloading,
resizing, and transferring the image to the image servers).
* The file upload component polls `/uploads/$id.json`, checking the
upload `status` until it returns `completed` or `error`.
* When the upload status is `completed`, the user is redirected to /uploads/$id.
* On the /uploads/$id page, the user can tag the upload and submit it.
* The upload form calls `POST /posts` to create a new post from the upload.
* The user is redirected to the new post.
This is the data model:
* An upload represents a set of files uploaded to Danbooru by a user.
Uploaded files don't have to belong to a post. An upload has an
uploader, a status (pending, processing, completed, or error), a
source (unless uploading from a file), and a list of media assets
(image or video files).
* There is a has-and-belongs-to-many relationship between uploads and
media assets. An upload can have many media assets, and a media asset
can belong to multiple uploads. Uploads are joined to media assets
through a upload_media_assets table.
An upload could potentially have multiple media assets if it's a Pixiv
or Twitter gallery. This is not yet implemented (at the moment all
uploads have one media asset).
A media asset can belong to multiple uploads if multiple people try
to upload the same file, or if the same user tries to upload the same
file more than once.
New features:
* On the upload page, you can press Ctrl+V to paste an URL and immediately upload it.
* You can save files for upload later. Your saved files are at /uploads.
Fixes:
* Improved error messages when uploading invalid files, bad URLs, and
when forgetting the rating.
Normalize artist group names following the same rules as artist other names.
This means artist group names now use underscores instead of spaces.
It also means extra space characters at the beginning and end of names
is stripped, and Unicode characters are normalized.
Fixes#4647, which was caused by users accidentally replacing group
names with a single space character when trying to remove a group.
Autotag non-web_source on posts that have a non-http:// or https:// URL.
Add a fix script to backfill old posts.
Syntactically invalid URLs are still considered web sources. For
example, `https://google,com` technically isn't a valid URL, but it's
not considered a non-web source.
When a banned artist tag is aliased into a nonbanned artist tag, move
the is_banned flag from the old artist entry to the new artist_entry.
Related to #4940. Fixes a case where a banned artist could lose the
banned status when it was moved.
Fix an `ActionView::Template::Error: undefined method 'verification_key'
for nil` error in the welcome_user mailer when a user signs up without
an email address.
Caused by the fact that we now render mail templates regardless of
whether the user has an email address, and then skip sending the email
only after the mail template is rendered.
Fix mailers to not attempt deliveries to invalid or nonexistent email
addresses. This usually happened when someone changed their email, and
we tried to send a confirmation email to a nonexistent address.
Remove the ability to lock a tag's category. Before a moderator could
lock a tag such that only an admin could change the tag's category.
Nowadays the ability to change a tag's category is based on the tag's
size. Members can change tag categories for tags with up to 50 posts,
and Builders can change categories for tags with up to 1000 posts.
Manually locking tags is not necessary.
We only had a few dozen locked tags, mostly random *_(cosplay) tags or
company name tags. Most of these are holdovers from moderators randomly
locking tags like ten years ago.
The `is_locked` field is still in the database, so it is still returned
by the /tags.json API, even though it is unused.
The problem was that we were stripping color profiles from thumbnails,
but we weren't setting `export_profile: "srgb"` to convert images to
sRGB first. This resulted in wrong colors for images with non-sRGB color
profiles, such as Adobe RGB.
The fix is to convert images to sRGB when possible, while leaving CMYK
and greyscale images alone. We leave CMYK images alone because we can't
convert CMYK to sRGB without losing color. We leave greyscale images
alone if they don't have a color profile, that way they stay as
one-channel greyscale (or two-channel greyscale, in case of alpha)
instead of being converted to three-channel sRGB. However, if a
greyscale image has a color profile, then we have to convert to sRGB,
otherwise the colors would be wrong when we strip the profile.
We also have to set the import profile, otherwise images with broken
embedded color profiles won't have a fallback profile and may get
incorrect colors. In this case we also have to be careful, because we
can't specify an sRGB fallback for greyscale or CMYK images.
Make the `order:random` metatag truly randomize the search. Add a
`random:N` metatag that returns up to N random posts, like what
`order:random` did before.
`order:random` now returns the entire search in random order. Before it
just returned a pageful of pseudorandom posts. This will be more
accurate for small searches, but slower for large searches. If
`order:random` times out, try `random:N` instead.
The `random:N` metatag returns up to N pseudorandom posts. This is
faster than `order:random` for large searches, but for small searches,
it may return less than N posts, and the randomness may be biased. Some
posts may be more likely than others to appear. N must be between 0 and
200.
Also, `/posts?tags=touhou&random=1` now redirects to `/posts?tags=touhou+random:N`.
Before the `random=1` param acted like a free `order:random` tag; now it
redirects to a `random:N` search, so it counts against your tag limit.
Allow admins to remove votes on posts. This is for fixing vote abuse.
Votes can be removed by going to the vote list on the /post_votes page,
or by clicking on a post's score, then using the "Remove" option in the
"..." dropdown menu next to the vote.
Votes are soft-deleted - they're marked as deleted in the database, but
not fully deleted. Removed votes are only visible to admins, not to
regular users. When a vote is removed by an admin, it leaves a mod
action.
Technically it's possible to undelete votes, but there's no UI for it.
* Add a gap between thumbnails on mobile.
* Adjust CSS for scores and vote buttons.
* Include "Private favorites" as an incentive on the user upgrade page.
* Fix vote buttons not being visible beneath thumbnails on mobile.
* Fix the "Show scores" link not preserving the current page number.
* Fix vote buttons being unintentionally enabled for all thumbnails by default.
* Fix banned and restricted users being able to favorite posts by
tagging them with `fav:self`.
* Fix search engines being able to crawl /posts?view=score pages.
* Fix broken tests.
Make upvotes public the same way favorites are public:
* Rename the "Private favorites" account setting to "Private favorites and upvotes".
* Make upvotes public, unless the user has private upvotes enabled. Note
that private upvotes are still visible to admins. Downvotes are still
hidden to everyone except for admins.
* Make https://danbooru.donmai.us/post_votes visible to all users. This
page shows all public upvotes. Private upvotes and downvotes are only
visible on the page to admins and to the voter themselves.
* Make votes searchable with the `upvote:username` and `downvote:username`
metatags. These already existed before, but they were only usable by
admins and by people searching for their own votes.
Upvotes are public to discourage users from upvoting with multiple
accounts. Upvote abuse is obvious to everyone when upvotes are public.
The other reason is to make upvotes consistent with favorites, which are
already public.
Add `upvotes:N`, `downvotes:N`, `order:upvotes`, `order:downvotes`,
`order:upvotes_asc`, `order:downvotes_asc` metatags.
In the API, the field is called up_score / down_score. Here it's called
`upvotes` and `downvotes` because this should be easier to understand
for end users.
Note that internally, `down_score` is negative. A post that matches
`downvotes:>5` will have down_score < -5 internally.
Fix certain artist commentaries for foundation.app containing scrambled
characters. Apparently caused by the Nokogiri HTML5 parser not handling
UTF-8 input correctly when the encoding isn't explicitly set to UTF-8.
