jobs: switch from DelayedJob to GoodJob.

Switch the ActiveJob backend from DelayedJob to GoodJob. Differences:

* The job worker is run with `bin/good_job start` instead of `bin/delayed_job`.
* Jobs have an 8 hour timeout instead of a 4 hour timeout.
* Jobs don't automatically retry on failure.
* Finishing jobs are preserved and pruned after 7 days.
This commit is contained in:
evazion
2022-01-04 13:52:08 -06:00
parent 21a9bb2c63
commit f4953549ae
15 changed files with 66 additions and 55 deletions

View File

@@ -9,30 +9,26 @@ later.
Jobs use the Rails Active Job framework. Active Job is a common framework that
allows jobs to be run on different job runner backends.
In the production environment, jobs are run using the Delayed Job backend. Jobs
are stored in the database in the `delayed_job` table. Worker processes spawned
by `bin/delayed_job` poll the table for new jobs to work.
In the production environment, jobs are run using the Good Job backend. Jobs
are stored in the database in the `good_jobs` table. Worker processes spawned
by `bin/good_job` poll the table for new jobs to work.
In the development environment, jobs are run with an in-process thread pool.
This will run jobs in the background, but will drop jobs when the server is
restarted.
There are two job queues, the `default` queue and the `bulk_update`. The
`bulk_update` queue handles bulk update requests. It has only one worker so that
bulk update requests are effectively processed sequentially. The `default` queue
handles everything else.
There is a very minimal admin dashboard for jobs at https://danbooru.donmai.us/delayed_jobs.
There is a very minimal admin dashboard for jobs at https://danbooru.donmai.us/jobs.
Danbooru also has periodic maintenance tasks that run in the background as cron
jobs. These are different from the jobs in this directory. See [app/logical/danbooru_maintenance.rb](../logical/danbooru_maintenance.rb).
jobs. These are different from the jobs in this directory. See
[app/logical/danbooru_maintenance.rb](../logical/danbooru_maintenance.rb).
# Usage
Start a pool of job workers:
```
RAILS_ENV=production bin/delayed_job --pool=default:8 --pool=bulk_update start
RAILS_ENV=production bin/good_job start --max-threads=4
```
# Examples
@@ -47,12 +43,12 @@ DeleteFavoritesJob.perform_later(user)
# See also
* [app/logical/danbooru_maintenance.rb](../logical/danbooru_maintenance.rb)
* [app/controllers/delayed_jobs_controller.rb](../controllers/delayed_jobs_controller.rb)
* [config/initializers/delayed_jobs.rb](../../config/initializers/delayed_jobs.rb)
* [app/controllers/jobs_controller.rb](../controllers/jobs_controller.rb)
* [config/initializers/good_job.rb](../../config/initializers/good_job.rb)
* [test/jobs](../../test/jobs)
# External links
* https://guides.rubyonrails.org/active_job_basics.html
* https://github.com/collectiveidea/delayed_job
* https://danbooru.donmai.us/delayed_jobs
* https://github.com/bensheldon/good_job
* https://danbooru.donmai.us/jobs

View File

@@ -3,15 +3,19 @@
# The base class for all background jobs on Danbooru.
#
# @see https://guides.rubyonrails.org/active_job_basics.html
# @see https://github.com/collectiveidea/delayed_job
# @see https://github.com/bensheldon/good_job
class ApplicationJob < ActiveJob::Base
class JobTimeoutError < StandardError; end
queue_as :default
queue_with_priority 0
around_perform do |_job, block|
CurrentUser.scoped(User.system, "127.0.0.1") do
ApplicationRecord.without_timeout do
block.call
Timeout.timeout(8.hours, JobTimeoutError) do
block.call
end
end
end
end

View File

@@ -5,13 +5,6 @@
#
# @see BigqueryExportService
class BigqueryExportJob < ApplicationJob
retry_on Exception, attempts: 0
# XXX delayed_job specific
def max_attempts
1
end
def perform(model:, **options)
BigqueryExportService.new(model, **options).export!
end

View File

@@ -3,13 +3,6 @@
# A job that sends notifications about new forum posts to Discord. Spawned by
# the {ForumPost} class when a new forum post is created.
class DiscordNotificationJob < ApplicationJob
retry_on Exception, attempts: 0
# XXX delayed_job specific
def max_attempts
1
end
def perform(forum_post:)
forum_post.send_discord_notification
end

View File

@@ -5,13 +5,6 @@
# @see {BulkUpdateRequestProcessor}
# @see {BulkUpdateRequest}
class ProcessBulkUpdateRequestJob < ApplicationJob
retry_on Exception, attempts: 0
# XXX delayed_job specific
def max_attempts
1
end
def perform(bulk_update_request)
bulk_update_request.processor.process!
end

View File

@@ -1,9 +0,0 @@
# frozen_string_literal: true
# A job that runs daily to delete all stale delayed jobs. Spawned by
# {DanbooruMaintenance}.
class PruneDelayedJobsJob < ApplicationJob
def perform
Delayed::Job.where("created_at < ?", 45.days.ago).delete_all
end
end

View File

@@ -0,0 +1,8 @@
# frozen_string_literal: true
# A job that runs daily to delete all stale jobs. Spawned by {DanbooruMaintenance}.
class PruneJobsJob < ApplicationJob
def perform
GoodJob::ActiveJobJob.where("created_at < ?", 7.days.ago).destroy_all
end
end

View File

@@ -12,7 +12,7 @@ module DanbooruMaintenance
end
def daily
queue PruneDelayedJobsJob
queue PruneJobsJob
queue PrunePostDisapprovalsJob
queue PruneBulkUpdateRequestsJob
queue PruneBansJob