Fix a bug where, when uploading a file from disk, if the filename was
too long, it wouldn't get word-wrapped and could break out of the
containing element.
Fix a bug where, when pasting an URL directly into the URL field, the
URL would be pasted twice, which would cause the upload to fail because
the URL was invalid.
* Fix broken upload tests.
* Fix uploads to return an error if both a file and a source are given
at the same time, or if neither are given. Also fix the error message
in this case so that it doesn't include "base" at the start of the string.
* Fix uploads to percent-encode any Unicode characters in the source URL.
* Add a max filesize validation to media assets.
Fix the upload page so that it shows similar images (IQDB matches) for
files uploaded from your computer. Before this only worked for files
uploaded from a source.
* Fix `UploadService is not a class` error.
* Update list of available job classes (remove UploadPreprocessorDelayedStartJob,
UploadServiceDelayedStartJob, add ProcessUploadJob).
Rework the upload process so that files are saved to Danbooru first
before the user starts tagging the upload.
The main user-visible change is that you have to select the file first
before you can start tagging it. Saving the file first lets us fix a
number of problems:
* We can check for dupes before the user tags the upload.
* We can perform dupe checks and show preview images for users not using the bookmarklet.
* We can show preview images without having to proxy images through Danbooru.
* We can show previews of videos and ugoira files.
* We can reliably show the filesize and resolution of the image.
* We can let the user save files to upload later.
* We can get rid of a lot of spaghetti code related to preprocessing
uploads. This was the cause of most weird "md5 confirmation doesn't
match md5" errors.
(Not all of these are implemented yet.)
Internally, uploading is now a two-step process: first we create an upload
object, then we create a post from the upload. This is how it works:
* The user goes to /uploads/new and chooses a file or pastes an URL into
the file upload component.
* The file upload component calls `POST /uploads` to create an upload.
* `POST /uploads` immediately returns a new upload object in the `pending` state.
* Danbooru starts processing the upload in a background job (downloading,
resizing, and transferring the image to the image servers).
* The file upload component polls `/uploads/$id.json`, checking the
upload `status` until it returns `completed` or `error`.
* When the upload status is `completed`, the user is redirected to /uploads/$id.
* On the /uploads/$id page, the user can tag the upload and submit it.
* The upload form calls `POST /posts` to create a new post from the upload.
* The user is redirected to the new post.
This is the data model:
* An upload represents a set of files uploaded to Danbooru by a user.
Uploaded files don't have to belong to a post. An upload has an
uploader, a status (pending, processing, completed, or error), a
source (unless uploading from a file), and a list of media assets
(image or video files).
* There is a has-and-belongs-to-many relationship between uploads and
media assets. An upload can have many media assets, and a media asset
can belong to multiple uploads. Uploads are joined to media assets
through a upload_media_assets table.
An upload could potentially have multiple media assets if it's a Pixiv
or Twitter gallery. This is not yet implemented (at the moment all
uploads have one media asset).
A media asset can belong to multiple uploads if multiple people try
to upload the same file, or if the same user tries to upload the same
file more than once.
New features:
* On the upload page, you can press Ctrl+V to paste an URL and immediately upload it.
* You can save files for upload later. Your saved files are at /uploads.
Fixes:
* Improved error messages when uploading invalid files, bad URLs, and
when forgetting the rating.
Add a join table that allows multiple media assets (images or videos) to
be attached to uploads. This is for a future ability to upload multiple
files at once.
Broken by the upgrade to webpacker-6.0.0.rc.6. Webpacker now defaults to
loading the Javascript bundle with `<script src="..." defer>`, which
means Javascript won't be loaded until after the page loads, which broke
the code that records view counts because it uses jQuery and jQuery
wasn't loaded yet.
Contrary to its name, `image-rendering: smooth` resulted in pixelated
edges when images were downscaled to fit the browser window. This only
affected Firefox because other browsers don't support `smooth`.
Disable font boosting on mobile. By default, when desktop mode is
enabled on mobile, mobile browsers will automagically increase the size
of text. Usually they do so poorly, making things like headers smaller
than body text, which breaks the layout.
Fixes regression in d6b1302e0.
Fix regression introduced in 0db20e0ca. Setting `format: false` on the
wiki pages resource disabled format negotiation on all wiki page routes,
not just the show page, which meant /wiki_pages.json no longer worked.
The fix to monkey patch the internal Rails method that parses the file
extension from the URL, and have it ignore everything but the .html,
.json, .js, and .xml extensions. This is really hacky and may break in
future Rails releases.
Fix it so that when a forum topic is deleted, all posts in the topic are
deleted too. Also make it so that when a forum topic is undeleted, all
posts in it are undeleted too.
Before when a topic was deleted, only the topic itself was marked as
deleted, not the posts inside the topic. This meant that when a spam
topic was deleted, the OP wouldn't be marked as deleted, so any
modreports against it wouldn't be marked as handled.
Also change it so that it's not possible to undelete a post in a deleted
topic, or to delete the OP of a topic without deleting the topic itself.
Finally, add a fix script to delete all active posts in deleted topics,
and to undelete all deleted OPs in active topics.
* Add ability to mark moderation reports as 'handled' or 'rejected'.
* Automatically mark reports as handled when the comment or forum post
is deleted.
* Send a dmail to the reporter when their report is handled.
* Don't show the report notice on comments or forum posts when all
reports against it have been handled or rejected.
* Add a fix script to mark all existing reports for deleted comments,
forum posts, or dmails as handled.
There are a lot of old artist entries with Japanese names. These names
are now invalid and these artist entries can't be edited because they
fail validation checks.
Add a fix script to delete all artist entries with non-ASCII names, and
rename them to `artist_1234`.
Normalize artist group names following the same rules as artist other names.
This means artist group names now use underscores instead of spaces.
It also means extra space characters at the beginning and end of names
is stripped, and Unicode characters are normalized.
Fixes#4647, which was caused by users accidentally replacing group
names with a single space character when trying to remove a group.
Only show changed URLs in the artist history, not unchanged URLs. Makes
the edit history more compact and easier to read by showing only the
things that changed on every edit.
Fix wiki pages like this returning 406 errors:
* https://danbooru.donmai.us/wiki_pages/rnd.jpg
Caused by Rails parsing the .jpg part as a file extension and trying to
return a JPEG in response. This happens deep in Rails' MIME negotiation
code, so it's hard to override. The fix is to pass `format: false` in
the route to disable all special handling of file extensions by Rails,
and then handle it ourselves in the controller. Ugly.
This only affected two tags: `rnd.jpg` and `haru.jpg`.
Remove all unnecessary rules from our CSS reset stylesheet.
Our CSS reset was based on a combination of Eric Meyer's CSS reset [1]
and Nicholas Gallagher's normalize.css [2]. Neither had been updated in
over 10 years, so they both contained a lot of unnecessary cruft. This
included workarounds for bugs in ancient versions of IE, and rules for
elements we don't use. Some of these rules had already been removed
upstream, but we never synchronized our copy of normalize.css with it.
This fixes some minor issues with vertical alignment of form inputs,
caused by unnecessary `vertical-align: middle` rules.
[1]: https://meyerweb.com/eric/tools/css/reset/
[2]: https://nicolasgallagher.com/about-normalize-css/
Adjust heading tags (h1, h2, h3, h4, h5, h6) to use margins instead of
padding so that margins collapse together. Adjust margin sizes to reduce
gaps between headers and paragraphs, and headers and subheaders.
Also adjust paragraph margins so that there's slightly less space
between paragraphs.
Autotag non-web_source on posts that have a non-http:// or https:// URL.
Add a fix script to backfill old posts.
Syntactically invalid URLs are still considered web sources. For
example, `https://google,com` technically isn't a valid URL, but it's
not considered a non-web source.
Fix the paginator not appearing when all posts on the page are hidden,
because of deleted posts, banned artists, censored tags, or non-safe
posts in safe mode. This prevented navigating to the next or previous
page.
Fix BigQuery export jobs failing with:
Google::Cloud::InvalidArgumentError: required: Bucket is requester pays
bucket but no user project provided.
Caused by changing the storage bucket to requester pays. The
`user_project` param must be set to true on requester pays buckets to
bill usage to the current project.
Add a 'Copy Link' action to forum posts and comments. This copies the
full link to the clipboard. The 'Copy ID' action copies just the DText
shortlink (comment #XXX or forum #XXX).
Set the MALLOC_CONF environment variable in the Docker image to tune the
Jemalloc configuration. Configuring Jemalloc to use two memory arenas
reduces memory fragmentation, and using background threads and low decay
times allows freed memory to be returned to the OS sooner.
Previously we set this environment variable at runtime in Kubernetes,
but baking it into the image is simpler.
* Add data-is-deleted, data-updater-id, data-topic-id, and
data-topic-is-deleted data attributes to forum posts.
* Change data-forum-post-id to data-id.
* Dim deleted forum posts like comments.
Add a `z=N` param to the URL when doing a search from the search box, or
when clicking a link in the tag list. This is to gather data on how
often the links in the tag list are used, and whether it's more common
to click tags in the tag list or to perform searches manually.
This is temporary and will be removed when enough data is collected.
Don't default the "Final source" field to the current source. This way
the default will be to set the source to the normalized replacement URL.
The "final source" field can be set to override this.
Fix the /posts index controller not logging the normalized search query
to NewRelic when the search failed, either because of a tag limit error,
a search timeout, or a RSS feed rate limit error.
Also don't log the number of search results when it's an API request or
failed search. This is to avoid doing a potentially slow full post count
when it's not otherwise needed.
When a banned artist tag is aliased into a nonbanned artist tag, move
the is_banned flag from the old artist entry to the new artist_entry.
Related to #4940. Fixes a case where a banned artist could lose the
banned status when it was moved.
Add foreign key constraints on all foreign keys on all tables.
These constraints are deferrable so that they're checked at the end of
the transaction, rather at the end of the statement. This is to reduce
lock duration and to allow for cyclic relationships.
Constraints are added in one migration then validated in another so that
the entire table isn't locked against reads and writes while the foreign
key constraints are being validated.
A few tables had invalid foreign keys. Add a fix script to fix these tables:
* A couple artist versions belonged to deleted artists.
* One dmail belonged to a deleted user.
* One forum topic visit belonged to that same deleted user.
* A few dozen note versions belonged to nonexistent posts. This came
from RaisingK moving notes to different posts years ago, back when it
was possible for users to set a note's post ID in the API.
* Some uploads had their parent ID set to 0.
Delete favorites that have an invalid post_id because they belong to an
expunged post.
This bug of not deleting favorites after a post is expunged was fixed
long ago, but old favorites were never cleaned up.
Fixes#4711: Some users have incorrect fav count.
Update the minimum browser requirement from browsers released since
2012, to browsers released in the last 5 years.
This means PostCSS won't prefix CSS properties like `flex` or other
properties supported by browsers since 2017. Babel also won't transpile
Javascript syntax that has been supported by browsers since 2017.
Remove a PostCSS plugin used to workaround bugs in the `flex` CSS
property. Most of these bugs are no longer relevant and only affected
ancient browsers like IE 10 or 11.
https://github.com/philipwalton/flexbugs
Don't include fallback CSS properties for CSS variables. Before we
generated CSS like this:
a.tag-type-1 {
color: #c00004;
color: var(--artist-tag-color);
}
Now we generate CSS like this:
a.tag-type-1 {
color: var(--artist-tag-color);
}
This means that support for CSS variables is now required for colors to
work properly. All major browsers have supported CSS variables since
2016-2017.
Disable the ability to click and drag thumbnails. This usually happened
by accident when you clicked on a thumbnail and accidentally dragged it,
especially during tag scripting.
Fix an open redirect exploit where if you went to <https://danbooru.donmai.us/login?url=//fakebooru.com>,
then after you logged in you would be redirected to https://fakebooru.com.
This was actually fixed by the upgrade to Rails 7.0. `redirect_to` now
raises an `UnsafeRedirectError` on redirect to an offsite URL. Before we
tried to prevent offsite redirects by checking that the URL started with
a slash, but this was insufficient - it allowed protocol-relative URLs
like `//fakebooru.com`.
Add a test case for protocol-relative URLs and return a 403 error on an
offsite redirect.
This returns a Server-Timing header on all HTTP responses, which
includes details on how long it took the server to render the response.
Browsers can show this timing information in the devtools. In Chrome, go
to the Network panel, then click a HTTP request, then click the Timing tab.
* Update framework files with `bin/rails app:update`.
* Update to use new Rails 7.0 default settings, except for a couple
things regarding new cookie and cache formats that would prevent us
from rolling back to Rails 6.1 if necessary.
Fix the paginator not detecting the first or last page correctly during
sequential pagination.
Caused by the fact that we fetch one more record than needed to detect
whether we're on the last page, then throw that record away by
overriding Rails' internal `records` method. An upstream refactoring
meant that the `size` method now counts the number of records *after*
the extra record is thrown away, where before it counted *before* the
extra record was thrown away.
Previously we overrode `ActiveRecord::Base#columns` to make it ignore
tsvector columns so they wouldn't show up in API responses. These
columns have been removed, so this is no longer necessary.
is_note_locked, is_rating_locked, and is_status_locked have been unused
since 126046cb6.
tag_index has been unused since 37a8dc5db.
fav_string has been unused since 165339236.
pool_string has been unused since 7d503f088.
evazion
evazion
