* Move search form to index page instead of a separate page.
* Merge creator + timestamp into one column.
* Add category column.
* Fix N+1 query issue.
Move the parsing for the [bur:<id>], [ta:<id>], [ti:<id>] pseudo tags to
the main parser in `DText.format_text`. This fixes a bug where wiki
links inside bulk update requests on the forum weren't properly
colorized because the text of the BUR was embedded after we scanned for
wiki links, not before.
This also ensures that tags inside bulk update requests will be recorded
in the dtext_links table, meaning that forum posts can be properly
searched by tags.
This incidentally means that these request pseudo tags can now be used
outside the forum.
Normally we skip doing page counts on index pages when there aren't any
search filters. This is on the assumption that most index pages have
more than 1000 pages (20,000 results), so it's not worth counting them
exactly. This isn't always true, so here we turn on full counts on
certain index pages known to be small.
* Add post/comment/forum vote counts to user profiles.
* Show uploaders on post votes index and allow searching by uploader.
* Show forum posters on forum votes index and allow searching by poster.
* Add unvote link to forum votes index.
* Only show unvote links to current user.
* Let gold users use upvote:self, downvote:self metatags to search for
their own votes.
* Don't let mods use upvote:<user>, downvote:<user> metatags to see
votes by other users. Only let admins see other users' votes.
* Add vote count to profile page.
* Move forum post vote tests from test/controllers to test/functional.
* Fix forum post vote tests to work with new routes.
* Fix obsolete wiki page tests dealing with updater_id.
* Remove the single alias and implication request forms. From now
on, bulk update requests are the only way to request aliases or
implications.
* Remove the forum topic ID field from the bulk update request form.
Instead, to attach a BUR to an existing topic you go to the topic then
you click "Request alias/implication" at the top of the page.
* Update the bulk update request form to give better examples for the
script format and to explain the difference between aliases and
implications.
Instead of sending IQDB the image url and letting it download the file,
download the file on Danbooru's end and send IQDB the downloaded file.
This fixes several issues:
* Some sites need the referer header set to avoid hotlink protection
when downloading the file. Danbooru knows how to deal with this but
IQDB doesn't.
* We need to enforce certain restrictions when downloading files,
including setting max filesize limits, setting max timeouts, and not
allowing downloads from forbidden IPs (to avoid SSRF attacks).
Danbooru knows how to handle these things but IQDB doesn't.
Fix the moebooru strategy to fallback to returning the image url if we
can't find the preview url. Fixes iqdb lookups failing in some cases
because the strategy didn't return a valid url for preview_url.
* Pick the largest character or copyright tags by post count. Previously
we picked the tags with the longest names, which was nonsensical.
* Remove tag cateogory logic from config file. We can't avoid hardcoding
some knowledge about tag categories here, so there's no point in trying.
This affects tab titles on post show pages as well as filenames in
downloaded images.
Drop the creator_id and updater_id fields from wiki pages. These fields
had several issues:
* The creator_id field was inconsistent with the wiki_page_versions
table. Apparently during the migration to Danbooru 2 in 2012-2013 the
creator_id field got reset to whoever last updated the wiki at that
point in time.
* Saving a wiki would set the updater_id even when nothing actually
changed. This also caused the updated_at timestamp to get bumped.
Because of this, anything that saved a wiki, including things like
creating aliases or implications, would bump the updater_id and
updated_at even though the wiki didn't actually change. This meant
these fields weren't consistent with the wiki_page_versions history.
Changes:
* Remove `creator_name` field from the /wiki_pages.json API.
* Remove creator name search option from /wiki_pages/search.
Add a dtext_links table for tracking links between wiki pages. This is
to allow for broken link detection and "what links here" searches, among
other uses.
DText was changed so that the .dtext-external-link class is now applied to all external links.
Previously it applied only to named links (ex: "google":[http://www.google.com]), not bare
links (ex: http://www.google.com).
https://fontawesome.com/how-to-use/on-the-web/other-topics/performance
The default version of Font Awesome uses Javascript to replace <i> tags
with dynamically generated SVG elements. This adds a lot of weight to
the Javascript bundle (at least 1MB+), even when using subsetting to
load only the icons we actually use. The web font version is less
featureful than the JS version, but much lighter weight.
* Fix the IQDB lookup to handle error messages returned by IQDB.
* Fix `undefined method `>=' for nil:NilClass` error when trying to upload
ugoiras. Using the preview image for IQDB lookups should be faster and
give the same results as the full image, plus it should work with things
like ugoiras and videos that IQDB can't handle.
Also add an image_url param so that API users can continue using the
full image if needed.
https://danbooru.donmai.us/forum_topics/9127?page=283#forum_post_160508
There was a recent outage that was caused by the read replica
(yukinoshita.donmai.us) being temporarily unavailable. The pg driver in
rails got hardstuck trying to connect to the replica, which brought down
the whole site. The app servers stopped responding and could only be
brought down with SIGKILL. Even try to boot the rails console didn't
work.
We only really used this to calculate tag counts inside Post.fast_count,
which wasn't really beneficial since the read replica is slower than the
main database.
Previously the search form on the /iqdb_queries page submitted directly
to the iqdb service (karasuma.donmai.us), which redirected back to
Danbooru with the search results.
This was different than API requests, which submitted to
/iqdb_queries.json which proxied the call to iqdb through Danbooru.
Because of this, searches on the /iqdb_queries page had different
behavior than API requests. Things like filesize limits and referrer
spoofing were handled differently.
Now searches on the /iqdb_queries page submit directly to Danbooru. This
is simpler and it means that API requests and HTML requests have the
same behavior.
Normally thumbnails have a fixed size of 154x154, but that's not always
desirable outside of the posts index because it creates empty gaps
around thumbnails.
* Add the source (twitter, pixiv, etc) and upload date ("X minutes ago")
to iqdb thumbnails.
* Link the filesize to the full file so you can compare files in new tabs.
* Link the similarity to a iqdb search so you can pivot your search to other posts.
* Change cutoffs on upload page to max 5 results, min. 20% similarity.
* Change cutoffs on standalone /iqdb_queries page to max 20 results, min. 0% similarity.
* /iqdb_queries.json: add `limit` and `similarity` params to change default cutoffs.
Remove the 'Similar' button next to the source field in the post edit
form. Removed for multiple reasons:
* It doesn't make sense to have to open the edit form to do a reverse
image search.
* The 'Similar' button tries to redownload the file from the source,
which has various problems: the source might have been deleted, it
might have been changed or revised, it might be a format that iqdb
can't handle (ugoira/webm/mp4), or it might otherwise not match the
the actual post.
* The 'Find similar' button already exists in the sidebar and it does
the right thing by using the preview image from Danbooru, which
avoids all the above issues.
Disable the assets pipeline (Sprockets). Sprockets errors out now after
upgrading to Sprockets 4 because of missing config files. We don't use
it any more after switching to Webpack, so we can disable it entirely.
Also disable a few more Rails features that we don't use (ActiveStorage,
ActionCable, ActionMailbox, ActionText).
This was used to discourage crawlers from crawling certain pages we
didn't want them to crawl, primarily post searches.
Remove because there are better ways to control crawling. Some of these
links weren't even visible to crawlers anyway. This lets us be
consistent about only applying rel="nofollow" to external links.
* Use underscores instead of spaces for tags in inline tag lists (upload
tags report, tooltips, modqueue, comments page).
* Allow long tags to word wrap. Fixes long sources not wrapping in the
uploads tag report. Also fixes very long tags that don't have
underscores not wrapping in the sidebar (ex: kuouzumiaiginsusutakeizumonokamimeichoujin_mika).
* Don't truncate long sources in the sidebar on the post show page. Word
wrap them instead.
* Word wrap long external links in general (mainly links in dtext).
* Turn sources into links on modqueue page.
DText is processed in three phases: a preprocessing phase, the regular
parsing phases, and a postprocessing phase.
In the preprocessing phase we extract all the wiki links from all the
dtext messages on the page (more precisely, we do this in forum threads
and on comment pages, because these are the main places with lots of
dtext). This is so we can lookup all the tags and wiki pages in one
query, which is necessary because in the worst case (in certain forum
threads and in certain list_of_* wiki pages) there can be hundreds of
tags per page.
In the postprocessing phase we fixup the html generated by the ragel
parser to add CSS classes to wiki links. We do this in a postprocessing
step because it's easier than doing it in the ragel parser itself.
* Parse the wiki page with the actual dtext parser instead of by hand.
This is so that wiki links inside things like [nodtext] or [code]
blocks are handled properly.
* Only include tags that exist and are nonempty. Don't include links to
dead pages or blank tags.
There are a handful of places where we need to strip markup from a piece
of dtext, primarily in <meta> description tags in the wiki. Currently
the dtext parser handles this by having a special mode where it parses
the text but doesn't output html tags. Here we refactor to instead parse
the text normally then strip out the html tags after the fact.
This is more flexible and allows us to simplify a lot of things in the
dtext parser. This also produces more readable output than before in
certain cases.
The swipe left gesture interfered with scrolling left and right, using
using pinch to zoom, and with copy and pasting text. This gesture wasn't
really necessary anyway, since the back button can always be used to go
back instead.
* Allow both xml and json authentication in sessions controller.
* Raise an exception if a login attempt fails so that a) we return a
proper error for json/xml requests and b) failed login attempts get
reported to NewRelic (for monitoring abuse).
Replace this common pattern in controllers:
@tags = Tag.search(search_params).paginate(params[:page], :limit => params[:limit], :search_count => params[:search])
with this:
@tags = Tag.paginated_search(params)
`search_count` is used to skip doing a full page count when we're not
doing a search (on the assumption that the number of results will be
high when not constrained by a search). We didn't do this consistently
though. Refactor to do this in every controller.
Previously the page-based (numbered) paginator would always count the
total_pages, even in API calls when it wasn't needed. This could be very
slow in some cases. Refactor so that total_pages isn't calculated unless
it's called.
While we're at it, refactor to condense all the sequential vs. numbered
pagination logic into one module. This incidentally fixes a couple more
bugs:
* "page=b0" returned all pages rather than nothing.
* Bad parameters like "page=blaha123" and "page=a123blah" were accepted.
Don't send the referer when hotlinking preview images on the
/uploads/new page. Improves user privacy and may bypass anti-hotlinking
protections in some cases.
Set the thumbnail height to auto instead of 154px so that there's not a
big empty gap beneath thumbnails that aren't 150px high.
This requires setting dimensions on the <img> tag itself so that the
image height is known before loading, otherwise the tooltip will be
mispositioned after the image loads in.
We set the min-width and min-height instead of the <img> width and
height properties because our calculated dimensions are sometimes
off-by-one compared to the actual dimensions. I'm not sure how libvips
calculates the thumbnail dimensions, but this avoids forcing a slightly
wrong aspect ratio, which produces ugly resizing artifacts.
* Cancel pending ajax requests when mousing out of the thumbnail.
Prevents multiple requests from piling up if the user moves in and out
of the thumbnail before the first request completes. This normally
isn't possible except during slowbooru.
* Show an error message if the ajax request fails unexpectedly.
Tells browsers not to send the Referer header when following external
links. Among other things, this bypasses Pixiv's anti-hotlinking
protection when opening direct image sources from Pixiv.
Refactor things to store information about the current user as data
attributes on the <body> tag rather than as <meta> tags. These <meta>
tags are now deprecated and will be eventually removed.
* Store all of the current user's API attributes as data attributes on
the <body> tag.
* Add `CurrentUser.data` for getting data from the <body> tag, and
use it instead of `Utility.meta`.
* Add `CurrentUser.update` for updating the current user's settings.
* Fix a bug with the user named "Anonymous" not being able to edit notes.
evazion
evazion
