Commit Graph

127 Commits

Author SHA1 Message Date
evazion
0f36bbf8d3 iqdb: update API client to use new version of IQDB.
Replace the old IQDB API client with a new client for the new forked
version of IQDB at https://github.com/danbooru/iqdb.

Changes:

* The /iqdb_queries endpoint now returns `hash` and `signature` fields.
  The `signature` is the full decoded Haar signature, while the `hash`
  is a encoded version of the signature.
* The /iqdb_queries endpoint no longer returns `width` and `height`
  fields in the response (these were always 128x128).
* We no longer need the IQDBs frontend server, now we talk to the IQDB
  instance directly.
* We no longer send add/remove image commands to IQDB through AWS SQS,
  now we send them to IQDB directly. They are sent in a delayed job so
  that if IQDB is down, uploading images is still possible, the add
  image commands will just get queued up.
* Fix a bug where regenerating an image's thumbnails didn't regenerate
  IQDB, because IQDB silently ignored add image commands when the image
  already existed in the database.
2021-06-16 05:36:24 -05:00
evazion
f65f24be0b docker: add cron service to compose file. 2021-05-25 01:16:59 -05:00
evazion
4439293bf1 newrelic: fix newrelic starting without license key.
Fix an issue where the New Relic agent always started in the production
environment, even when a license key wasn't configured.

Also make the New Relic agent log to stdout instead of log/newrelic_agent.log.
2021-05-24 21:58:01 -05:00
evazion
c22df03804 Move script/delayed_job to bin/delayed_job.
This has been the recommended location since Rails 4.
2021-05-24 17:38:56 -05:00
evazion
d825bb144b docker: add rake task for building Docker image.
Add a Rake task for building a Docker image. Usage:

* bin/rails danbooru:docker:build
* sudo docker run --rm -it danbooru
2021-05-02 17:42:11 -05:00
evazion
189adc683f config: set default database connection settings.
Set sensible defaults for connecting to the database. By default, we try
to connect to the `danbooru2` database running on localhost as the
`danbooru` user. These are the defaults recommended by the install
guide.

If you need to change the database settings, set DATABASE_URL in
.env.local or on the command line:

   # .env.local
   DATABASE_URL=postgresql://danbooru:password@localhost/danbooru2

   # command line
   $ DATABASE_URL=postgresql://danbooru:password@localhost/danbooru2 bin/rails server

This eliminates the need to copy script/install/database.yml.templ to
config/database.yml during installation and during deployment. This is
so that Danbooru works out of the box without extra configuration. In
particular, this is so that we can run Danbooru in a Docker container
without having to set DATABASE_URL.
2021-03-23 02:29:45 -05:00
evazion
c9ca5c0598 rake: fix images:manifest failing on file permission errors.
Fix images:manifest failing with an exception when a file can't be
opened because we don't have permission to access the file.
2021-03-20 16:15:16 -05:00
evazion
048171fe29 rake: fix images:manifest task to handle corrupted files. 2021-03-20 05:30:06 -05:00
evazion
6879fe73db rake: add task for checking image md5s.
Add a rake task for checking the md5s and size/width/height of all
uploaded files. Generates a JSON manifest file.

Usage: bin/rake images:manifest
2021-03-20 02:46:28 -05:00
evazion
fccdcbe7cc simple form: regenerate config with new defaults.
Regenerated with `bin/rails generate simple_form:install`.
2020-06-25 16:28:09 -05:00
evazion
5fc98eb35f rake: add image preview/sample regeneration script. 2020-05-23 23:51:23 -05:00
evazion
79aff89573 maintenance: prune approvers monthly instead of weekly.
* Move approver pruning from a weekly task to a monthly task.
* Move weekly maintenance tasks from Wednesdays at 1:30am to Sundays at
  midnight (UTC).
2020-01-13 13:23:22 -06:00
evazion
7da85a4eb7 rake: add images:backup rake task. 2019-12-30 23:00:53 -06:00
evazion
fe0bb1e602 rake: remove unused image tasks. 2019-12-30 23:00:53 -06:00
evazion
87fe3c63fb capistrano: fixup delayed jobs deployment. 2019-12-30 00:12:51 -06:00
evazion
3bcc503cf7 BURs: process bulk updates in dedicated job queue.
Add a dedicated queue for bulk update requests and process it using a
single worker. This prevents bulk updates from consuming all available
workers and preventing other job types from running.

This also effectively serializes bulk updates so that they're processed
one-at-a-time instead of in parallel. This will be slower overall but
may avoid some of the issues with indeterminate update order under
parallel updates.
2019-12-27 12:24:51 -06:00
evazion
309821bf73 rubocop: fix various style issues. 2019-12-22 21:23:37 -06:00
evazion
316690f393 storage manager: remove S3 backend.
Remove in favor of the rclone S3 backend.
2019-12-13 02:14:14 -06:00
evazion
65e53b86b3 Drop support for /cache/tag.json.
Drop support for https://danbooru.donmai.us/cache/tags.json. This was a
nightly dump of the tags table that was originally added in #1012. It
was never documented and never really used except for by the DanbooruUp
extension.
2019-08-29 00:52:23 -05:00
evazion
fd639e3c81 capistrano: lower downtime window for delayed jobs. 2019-08-22 16:56:42 -05:00
evazion
39d70ae5cf capistrano: simplify unicorn:terminate task.
Use the builtin task provided by capistrano3-unicorn. Ours failed with
invalid pidfile errors in some cases.
2019-08-15 12:51:51 -05:00
evazion
5f1226ca92 Rename maintenance.rb -> danbooru_maintenance.rb.
Fixes random test failures caused by ambiguous constant lookup issues
(the `Maintenance` module name was used in multiple conflicting places).
2019-08-15 00:36:25 -05:00
evazion
5acc75d21b capistrano: fix sleep calls.
Minor optimization for `cap production deploy --dry-run`. Sleep on the
server instead of in the client.
2019-08-12 18:22:44 -05:00
evazion
072885bbf8 delayed jobs: run dj workers under systemd (fix #4108). 2019-08-10 02:57:32 -05:00
evazion
bfffb57c13 capistrano: clean up unicorn:terminate task.
Do a SIGQUIT (graceful shutdown, allow workers to finish current
request) before doing a SIGTERM (immediate shutdown).
2019-08-07 22:36:12 -05:00
evazion
c225d54de1 capistrano: add deploy:rolling task (fix #4123). 2019-08-07 22:11:07 -05:00
evazion
6e54c37b04 maintenance: add rake tasks for maintenance jobs. 2018-10-09 20:11:24 -05:00
Albert Yi
897d3d3a99 add production2 env for testing new servers 2018-10-03 11:23:26 -07:00
Albert Yi
762dc3da24 Refactor sources 2018-08-24 12:10:51 -07:00
Albert Yi
c9fe91c74b fix base dir for image distribute rake task 2018-04-12 10:48:53 -07:00
Albert Yi
03e2a2ca3b add rake task for distributing images using storage managers 2018-04-12 10:40:24 -07:00
evazion
4b5129d447 lib/tasks/images.rake: fix bad ref to danbooru_image_resizer.
Fixup for efeaad25.
2017-12-18 18:08:41 -06:00
evazion
efeaad25ca Move lib/danbooru, lib/danbooru_image_resizer to app/logical. 2017-12-18 12:07:52 -06:00
r888888888
ec81b87766 include paginator in delayed::job 2017-12-13 17:15:25 -08:00
evazion
1c5c3b5f9f Include paginator extensions in ApplicationRecord. 2017-11-26 15:34:44 -06:00
r888888888
c2029b62b5 delayedjob fixes 2017-10-10 15:49:34 -07:00
r888888888
c23ea7f7b2 dont upload cropped thumbnails 2017-10-10 13:00:09 -07:00
r888888888
75f87bcc8e potential fix for #3290 2017-09-13 13:00:20 -07:00
r888888888
b9b38ca25e tweak css for responsive 2017-09-06 13:32:32 -07:00
evazion
ba408c8718 simple_form: regen config files (#2913).
Regens simple form config using `rails generate simple_form:install`.
Seems to fix #2913 (erroneous minlength validations being enforced as of
simple_form 3.4.0).
2017-03-03 17:21:37 -06:00
r888888888
6c9d5e4f9a remove advertisement code 2017-02-21 13:35:22 -08:00
r888888888
651dad473d fixes #2884: Comment unvoting is broken 2017-02-14 10:35:27 -08:00
r888888888
e5cd79eddd remove preview/large file upload rake tasks 2017-02-08 16:53:43 -08:00
r888888888
f0a8598ba2 update aws gem, stub in rake tasks for s3 2017-02-08 15:06:54 -08:00
r888888888
b57b54113f add cdn hosted flag to posts 2017-02-07 17:26:16 -08:00
Albert Yi
9a75a05c25 move newrelic + deploytags gem to production env, rename cap tasks to .rake 2016-12-06 15:47:21 -08:00
r888888888
8abd94b02d add kill step to unicorn cap 2016-08-02 22:14:50 -07:00
r888888888
7cf2dbc15b terminate unicorn procs on deploy 2016-08-02 22:12:14 -07:00
r888888888
d90fd6c46a force unicorn to terminate before restarting 2016-08-02 21:08:41 -07:00
r888888888
4fea400b79 fixes #2610: Strip image metadata when creating thumbnails and samples 2016-06-15 11:38:35 -07:00