Commit Graph
100 Commits
Author SHA1 Message Date
evazion da84e3a2f2 twitter: replace twitter gem with our own API client.
The twitter gem had several problems:

* It's been unmaintained for over a year.
* It pulled in a lot of dependencies, many of which were outdated. In
  particular, it locked the `http` gem to version 3.3, preventing us
  from upgrading to 4.2.
* It raised exceptions on normal error conditions, like for deleted
  tweets or suspended users, which we really don't want.
* We had to wrap it to provide caching.

Changes:

* Fixes #4226 (Exception when creating new artists entries for suspended
  Twitter accounts)
* Drop support for scraping images from summary cards. Summary cards
  are the previews you get when you link to a website in a tweet. These
  preview images aren't always the best image.
2019-12-13 17:27:03 -06:00
evazion 0b556ece1c Add Danbooru::Http client.
This is a wrapper around the http.rb gem that provides caching.
2019-12-13 15:32:21 -06:00
evazion ddb0e4d3ce config: refactor secret_key_base initialization. 2019-12-13 04:21:04 -06:00
evazion 6382aec85e config: update to Rails 6.0 defaults. 2019-12-13 04:21:04 -06:00
evazion fc5bce5595 redis: bump timeout settings. 2019-12-13 03:13:17 -06:00
evazion ea15b525f8 config: clean up filter_parameters initializer. 2019-12-13 03:05:46 -06:00
evazion a4e587aa6a config: refactor email settings.
* Add `Danbooru.config.mail_delivery_method` option.
* Remove `Danbooru.config.aws_ses_enabled?` option.
* Replace `Danbooru.config.aws_ses_options` with `Danbooru.config.mail_settings`.
2019-12-13 03:00:12 -06:00
evazion 33e3754c2d Update gems. 2019-12-13 02:14:20 -06:00
evazion 316690f393 storage manager: remove S3 backend.
Remove in favor of the rclone S3 backend.
2019-12-13 02:14:14 -06:00
evazion 92aca9ed2d storage manager: remove fog backend.
Remove in favor of rclone.
2019-12-13 01:45:43 -06:00
evazion 8e93b0fc6c /tags: fix link to [[help:tags]] in subnav menu. 2019-12-11 18:18:12 -06:00
evazion 99a1a408cd /artists/banned, /favorites: fix redirects for non-html requests.
ref: https://danbooru.donmai.us/forum_topics/9127?page=287#forum_post_161549
2019-12-11 02:27:41 -06:00
evazion afea704b55 storage manager: add rclone backend. 2019-12-11 01:44:23 -06:00
evazion 2e6486ab5f BURs: fix exception in category change requests for new tags.
`category blah -> character` failed when `blah` didn't exist yet.
2019-12-10 15:17:10 -06:00
evazion 2f27a4eddd image resizer: fix incompatibility with libvips-8.8.
In libvips-8.8 the `auto_rotate` option was replaced with `no_rotate`.
Using `auto_rotate` in libvips-8.8 fails, so we need a version check
here.
2019-12-10 14:04:40 -06:00
evazion 90f64eb407 image resizer: remove shell fallback for old versions of libvips.
Remove fallback for old versions of Debian. Debian stable now provides
libvips 8.7.
2019-12-10 14:04:40 -06:00
evazion 4b426ec5b9 storage manager: fix b2 storage bugs.
* B2 doesn't allow the path to start with a '/' character.
* When storing the file, we have to rewind the file pointer to make sure
  we get the whole file.
2019-12-10 14:04:40 -06:00
evazion e4f4326982 storage manager: add backblaze b2 backend. 2019-12-10 02:04:32 -06:00
evazion 6b165a90f8 tags: fix negative tag counts not being regenerated. 2019-12-10 01:43:05 -06:00
evazion 6196f7ba67 BURs: fix exception when estimating post counts of category changes.
Fix an error in operator precedence:

> sum + Tag.find_by_name(token[1]).try(:post_count) || 0

This was treated as `(sum + X) || 0` not `sum + (X || 0)` as intended.
This failed when X was nil.
2019-12-10 01:43:05 -06:00
evazion e906b902d8 circleci: reduce parallelism
Fix circleci builds not running because parallelism was set too high (free plan is limited to 4x).
2019-12-08 00:43:39 -06:00
evazion f24220982a seo: don't noindex /posts?tags=order:rank. 2019-12-07 22:18:58 -06:00
evazion fdffd3bdcb BURs: fix "This request will be attached to" link.
ref: https://danbooru.donmai.us/forum_topics/9127?page=287#forum_post_161392
2019-12-07 20:09:03 -06:00
evazion 5394dbc3d2 tagging: lower happy/sad tag count thresholds.
Lower the frowny face tag threshold from 25 to 10 tags and the happy
face tag threshold from 35 to 20 tags.
2019-12-07 19:03:40 -06:00
evazion 5919b825d7 seo: don't hide the front page from web crawlers.
Don't mark the front page (empty tag search) as noindex.
2019-12-07 18:35:53 -06:00
evazion 81f7d14fa3 uploads: bump iqdb similarity threshold 65% -> 70%. 2019-12-07 11:24:39 -06:00
evazion ea718122da Fix #4219: Deviantart broke download urls when not logged in. 2019-12-06 22:54:08 -06:00
evazion 1c3b96dc4e deviantart: fix non-downloadable posts (#4219).
Adapted from https://github.com/lllusion3469/danbooru/commit/e51e1aa7ddb73e92541bfbcab5c2f8c0359a07dc.
2019-12-06 22:54:08 -06:00
evazion 2bffad9311 BURs: credit tag edits to DanbooruBot.
Credit tag edits that are performed as part of an alias, implication, or
mass update to DanbooruBot instead of the admin who approved the BUR.
2019-12-06 22:54:08 -06:00
evazion 620b81e7b9 Fix #4218: Can't install because Rails assets manifest is missing. 2019-12-06 02:44:10 -06:00
evazion 2be9c113e2 Fix #4222: unrecognized configuration parameter "default_table_access_method"
This option was introduced in Postgres 12 and isn't backwards compatible
with older Postgres versions.
2019-12-06 02:32:15 -06:00
evazion 669ba6fc8d upgrades: fix post-upgrade landing page.
* Fix the post-upgrade message showing the gift message instead of the
  normal upgrade message.

* Link back to Danbooru so people don't stay on Safebooru and get
  confused when they can't see hidden posts.
2019-12-06 02:26:11 -06:00
evazion f355503413 tests: fix recommended posts tests. 2019-12-06 01:33:52 -06:00
evazion 0c483cad15 config: fix references to Danbooru.config.app_name.
Danbooru.config.app_name is either Safebooru or Danbooru. Fix some
places where it should always be Danbooru.
2019-12-06 01:33:52 -06:00
evazion 06078ef172 danbooru_default_config.rb: remove unused config params. 2019-12-06 01:33:52 -06:00
evazion 5cdccf4b93 Update gems. 2019-12-06 01:33:52 -06:00
evazion 0a5d8f57d5 ruby: bump ruby version to 2.6.5. 2019-12-05 22:32:45 -06:00
evazion cb45aa6fe6 Remove unused automod code.
The DynamoDB update job was disabled a few months ago. Remove it
completely.
2019-12-03 10:45:18 -06:00
evazion a559278953 notes: fix scrollbars appearing when using certain fonts. 2019-12-02 21:14:41 -06:00
evazion 5d1a1cc87e Change github repo url to https://github.com/danbooru/danbooru.
The danbooru repo was moved to https://github.com/danbooru.
2019-12-02 17:33:40 -06:00
evazion 008e97a787 capistrano: reduce delayed job worker count (16 -> 12). 2019-12-02 16:57:26 -06:00
evazion 56a2fae26f Revert "Temp disable account upgrades."
This reverts commit c4cdba0874.
2019-12-02 16:53:06 -06:00
evazion 86a09a322f cloudflare: use new api token auth scheme.
Use Cloudflare's new API token authentication method to talk to
Cloudflare. This lets us use API tokens with minimal privileges set.

https://blog.cloudflare.com/api-tokens-general-availability/
2019-12-02 16:45:32 -06:00
evazion 27a1a90cf3 recommendations: allow filtering recs by tags. 2019-12-02 03:08:01 -06:00
evazion 5791f4e805 recommendations: show favcount beneath recommended posts.
* Add favorite count beneath recommended posts. Clicking the favcount
  loads more recommended posts like that post.
* Increase number of recommendations shown on post show page.
2019-12-02 03:08:01 -06:00
evazion efda9f37e1 recommendations: add search form, allow searching by username. 2019-12-02 02:26:21 -06:00
evazion ae46f7a665 recommendations: hide recommendations for users with private favorites.
Don't allow users to view recommendations for users who have private
favorites.
2019-12-01 19:07:55 -06:00
evazion fa985f08ab recommendations: filter out own uploads and favorites.
Filter out the user's own uploads and favorites from their
recommendations.

Note that in most cases a user's top-N recommendations will be things
they've already favorited. If a user has 10,000 favorites, most of their
top 10,000 recommendations will be their own favorites, so we have to
generate a little more than 10,000 recommendations to be sure they won't
all be filtered out.

In other words, the more favorites a user has, the more recommendations
we have to generate. The upper bound is clamped to 50,000 for
performance reasons. If a user has more favorites than this we may not
be able to find any recommendations for them.
2019-12-01 19:03:26 -06:00
evazion 41b30fc64c recommendations: open user recommendations to all users.
* Open recommendations to all users (not just gold).
* Show recommendations on all posts (not just posts after 2017).
* Allow users to browse recommendations for other users.
* Increase number of recommended posts returned.
* Change endpoints to /recommended_posts?user_id=1234 and
  /recommended_posts?post_id=1234 and add json/xml support.
2019-12-01 00:44:04 -06:00
evazion c4cdba0874 Temp disable account upgrades. 2019-11-27 16:20:42 -06:00
evazion c0006b0b9a dtext: fix link colors bleeding through spoiler tags (again).
Caused by upgrading webpacker in f8e109f6d. This changed the load order
of the CSS files, which was significant because the rules for tag colors
and for spoiler tags had equal specificity.
2019-11-26 21:09:30 -06:00
evazion f8e109f6dc Update gems and yarn packages.
Remove *.scss.erb loader because it's now supported natively by
webpacker.
2019-11-26 01:58:57 -06:00
evazion 980a4b82b9 notes: adjust line-height for new fonts.
`line-height: normal;` produces large gaps between lines when using
non-default font families or font sizes. `line-height: 1.25;` reduces
these gaps.
2019-11-26 01:27:41 -06:00
evazion b43008a506 notes: add multiple new fonts (#4212).
Add the following fonts:

* Comic (Comic Relief)
* Slab Sans (Anton)
* Slab Serif (Rokkitt)
* Formal Serif (Lora)
* Formal Cursive (Petit Formal Script)
* Print (Kalam)
* Hand (Indie Flower)
* Narrow (Archivo Narrow)
* Blackletter (Unifraktur Maguntia)

Add the following font aliases:

* Comic Sans MS is aliased to Comic Relief.
* Arial Narrow is aliased to Archivo Narrow.
* Rockwell is aliased to Rokkitt.
* Impact is aliased to Anton.
2019-11-26 01:27:41 -06:00
evazion 761cbc47c1 notes: fix word breaks inside links.
https://danbooru.donmai.us/forum_topics/9127?page=286#forum_post_161140
2019-11-25 02:52:40 -06:00
evazion aaab527baa Move Curated pool updater to Danbooru.
* Move the Curated pool updater from Reportbooru to Danbooru.

* Change the process for selecting curated posts. Previously it was
  every post from the last week with at least three supervotes. This was
  flawed because it included both super-upvotes and super-downvotes. Now
  it's the top 100 posts from the last week, ordered from most super-upvoted
  to least.
2019-11-23 18:19:08 -06:00
evazion b489acfe00 Merge pull request #4216 from azisyus/postgresql-extension-build-tool
necessary package to build pgsql extension
2019-11-21 21:40:32 -06:00
evazion 060bf1a64b Update .gitignore. 2019-11-19 00:16:40 -06:00
evazion fd29f842fd db: drop keeper_data from posts. 2019-11-19 00:13:24 -06:00
evazion 839b9cf32f tests: fix session loader tests. 2019-11-19 00:06:20 -06:00
evazion 1ce9eff0f5 views: convert /notes to table builder. 2019-11-19 00:06:17 -06:00
evazion 5afac29693 views: convert /tags to table builder. 2019-11-17 22:35:53 -06:00
evazion be5df37328 views: add table builder abstraction. 2019-11-17 22:35:53 -06:00
evazion 43d0955b61 Fix #4065: Sometimes normalization brakes twitter intent links.
Fix incorrectly normalized Twitter intent URLs:

    url: https://twitter.com/intent/user?user_id=881080941443239936
    normalized_url: http://twitter.com/intent/

to

    url: https://twitter.com/intent/user?user_id=881080941443239936
    normalized_url: http://twitter.com/intent/user?user_id=881080941443239936/

Fixes about ~1300 incorrectly normalized urls in the production db.
2019-11-17 19:08:07 -06:00
evazion 91d1817abf artists: don't return domains in /artists/{id}.xml.
Don't return the `domains` field in /artists/{id}.{json,xml}. Fixes a
failure in /artists/{id}.xml:

    https://danbooru.donmai.us/artists/156646.xml

    <result success="false">
        undefined method `domains' for #<ArtistUrl:0x00005566dd340af0> Did you mean? DomainName
    </result>

`to_xml` passes down the `methods` param to all nested models, which
doesn't work.
2019-11-17 18:24:06 -06:00
evazion 320ff01e07 login: remove 'remember' checkbox; make session cookies permanent.
Remove the "Remember" checkbox from the login page. Make session cookies
permanent instead. Phase out legacy `user_name` and `password_hash` cookies.

Previously a user's session cookies would be cleared whenever they
closed their browser window, which would log them out of the site. To
work around this, when the "Remember" box was checked on the login page
(which it was by default), the user's name and password hash (!) would
be stored in separate permanent cookies, which would be used to
automatically log the user back in when their session cookies were
cleared. We can avoid all of this just by making the session cookies
themselves permanent.
2019-11-17 17:50:23 -06:00
evazion 9b893db640 autocomplete: fixup uses of tag_aliases.post_count.
Fixup for 64728c89d.
2019-11-17 16:03:56 -06:00
evazion d1689f895e tests: fixup /artists/banned endpoint test. 2019-11-17 14:41:09 -06:00
evazion 6e403fddb6 alias/implications: fixup uses of creator_ip_addr.
Fixup for a2ea2a65a. Use 127.0.0.1 rather than the creator's original IP
for any edits performed as part of an alias or implication.
2019-11-17 14:39:46 -06:00
evazion e29d7162a4 autocomplete: fixup limit param.
Fixiup for 378559ff4. Fix case when limit param is nil.
2019-11-17 14:39:46 -06:00
evazion 64728c89d8 db: remove tag_aliases.post_count column.
This was only used in autocomplete, but it was unnecessary here because
we could already get the post count from the tags table.
2019-11-17 14:39:41 -06:00
evazion a2ea2a65a1 db: drop IP addresses from certain tables.
Don't track IP addresses for post appeals, post flags, tag aliases, tag
implications, or user feedbacks. These things are already tightly
limited. We don't need IPs from them to detect sockpuppets.
2019-11-17 02:45:28 -06:00
evazion 72f17fd1de Fix #3534: Remove Janitor Trials. 2019-11-17 02:10:24 -06:00
evazion 1ae971269c db: drop tag subscriptions table. 2019-11-17 01:52:36 -06:00
evazion fedb525305 notes: add fallback font for comic sans (#4212). 2019-11-16 22:43:16 -06:00
evazion 364ecfe68f db: drop unused columns from users table.
Note that the password_hash column was replaced by bcrypt_password_hash
in 2013, but the old password_hash column was never dropped.
2019-11-16 19:34:30 -06:00
evazion 90713ef94c db: fix more inconsistencies between structure.sql and production db.
Fix various cases where db/structure.sql either didn't match the
production db or didn't match the db structure produced when running
migrations from a fresh install.
2019-11-16 16:35:04 -06:00
evazion 9248266659 Fix #4213: Empty Last IP link. 2019-11-16 10:39:57 -06:00
evazion 13f26c81d8 Remove unused files. 2019-11-15 23:59:56 -06:00
evazion 3746c9890a db: resync db/structure.sql. 2019-11-15 23:36:31 -06:00
evazion 529fef7224 db: set timestamps to not null. 2019-11-15 22:44:22 -06:00
evazion dad8114c97 db: fix index inconsistencies with production db. 2019-11-15 22:44:13 -06:00
evazion 0726e4346a migrations: fix old migration not removing posts.uploader_ip_addr index.
Bug: running migrations on a fresh database failed when we got to a
migration adding a posts.uploader_ip_addr index on 2019-11-11. It failed
because the index already existed.

The index already existed because we used to have it at one point, until
it was removed by another migration on 2015-01-20. This migration didn't
correctly remove the index though, because it tried to remove a
posts.source index first, which failed because this index didn't exist
(it probably existed only in production). This error was swallowed,
causing the migration to silently skip removing the posts.uploader_ip_addr
index, which eventually blew up when we tried to add the index again
later on.
2019-11-13 02:06:41 -06:00
evazion 378559ff47 autocomplete: make number of results configurable (partly).
Add `Danbooru.Autocomplete.MAX_RESULTS` param that controls the number
of results returned by autocomplete.

This does not work with tag autocomplete for Builders. Builder
autocomplete still needs to be refactored and unified with Member
autocomplete.

Also fix a bug in the /saved_searches/labels endpoint where the limit
param wasn't respected.
2019-11-13 01:47:41 -06:00
evazion 89c5ac73af autocomplete: switch to async/await. 2019-11-13 01:47:41 -06:00
evazion 2d1eabbe5f autocomplete: factor out clearing of menu bindings. 2019-11-13 01:47:37 -06:00
evazion 682aa220b8 /ip_addresses: add to sitemap; add IP info links. 2019-11-12 20:51:48 -06:00
evazion f9881e5414 autocomplete: add saved search label autocomplete in more places. 2019-11-12 20:24:24 -06:00
evazion cc98f3b7cc Update gems. 2019-11-12 19:05:32 -06:00
evazion da8fbef5d6 /ip_addresses: add missing created_at, uploader_ip_addr indexes.
Add some missing indexes used by the /ip_addresses page.
2019-11-11 21:03:07 -06:00
evazion 73c02021a1 /ip_addresses: allow searching multiple IPs at once.
Fixes #4208. Allow searching for multiple IPs or user IDs. Separate
using spaces or commas.
2019-11-11 21:03:07 -06:00
evazion 68e1140b0d /ip_addresses: allow grouping IPs by subnet. 2019-11-11 21:03:07 -06:00
evazion bf6bb94702 Add new IP address search page.
Add a new IP address search page at /ip_addresses. Replaces the old
search page at /moderator/ip_addrs.

On user profile pages, show the user's last known IP to mods. Also add
search links for finding other IPs or accounts associated with the user.

IP address search uses a big UNION ALL statement to merge IP addresses
across various tables into a single view. This makes searching easier,
but is known to timeout in certain cases.

Fixes #4207 (the new IP search page supports searching by subnet).
2019-11-11 21:03:07 -06:00
evazion 2aac42b112 Add show actions for note, artist, artist commentary versions.
Add these endpoints:

* /note_versions/1234
* /artist_versions/1234
* /artist_commentary_versions/1234

This is so the /ip_addresses listing can link to these endpoints.
2019-11-11 12:56:39 -06:00
evazion 9cf5b1f01c dtext: add dtext cheatsheet popup. 2019-11-10 14:48:17 -06:00
evazion 69f005a89c gems: remove cityhash gem.
This gem uses a native extension that requires a C++ compiler to build.
Removing this gem removes the need to have a C++ toolchain to install Danbooru.
2019-11-09 14:42:30 -06:00
evazion 76674597f1 Upgrade to Rails 6.0.1. 2019-11-06 21:57:32 -06:00
evazion a92c24f093 wiki pages: add undelete link in navbar.
Also redirect back to the wiki page after deleting or undeleting it,
rather than redirecting to the index.
2019-11-06 21:57:32 -06:00
evazion 70b6525e69 wiki pages: fix error when force renaming wiki page.
Bug: force renaming a wiki page fails. Fix: use the wiki page id instead
of the title in the update action.
2019-11-06 21:56:21 -06:00
evazion 19549e2c12 css: fix overlong parent/child preview boxes on post show page.
Bug: if a post had a large number of child posts, the child post preview
box would expand beyond the width of the screen.

Setting `min-width: 0` makes the flexbox column shrink instead of
expand.
2019-11-05 20:58:07 -06:00