jobs: switch from DelayedJob to GoodJob.
Switch the ActiveJob backend from DelayedJob to GoodJob. Differences: * The job worker is run with `bin/good_job start` instead of `bin/delayed_job`. * Jobs have an 8 hour timeout instead of a 4 hour timeout. * Jobs don't automatically retry on failure. * Finishing jobs are preserved and pruned after 7 days.
This commit is contained in:
@@ -9,30 +9,26 @@ later.
|
||||
Jobs use the Rails Active Job framework. Active Job is a common framework that
|
||||
allows jobs to be run on different job runner backends.
|
||||
|
||||
In the production environment, jobs are run using the Delayed Job backend. Jobs
|
||||
are stored in the database in the `delayed_job` table. Worker processes spawned
|
||||
by `bin/delayed_job` poll the table for new jobs to work.
|
||||
In the production environment, jobs are run using the Good Job backend. Jobs
|
||||
are stored in the database in the `good_jobs` table. Worker processes spawned
|
||||
by `bin/good_job` poll the table for new jobs to work.
|
||||
|
||||
In the development environment, jobs are run with an in-process thread pool.
|
||||
This will run jobs in the background, but will drop jobs when the server is
|
||||
restarted.
|
||||
|
||||
There are two job queues, the `default` queue and the `bulk_update`. The
|
||||
`bulk_update` queue handles bulk update requests. It has only one worker so that
|
||||
bulk update requests are effectively processed sequentially. The `default` queue
|
||||
handles everything else.
|
||||
|
||||
There is a very minimal admin dashboard for jobs at https://danbooru.donmai.us/delayed_jobs.
|
||||
There is a very minimal admin dashboard for jobs at https://danbooru.donmai.us/jobs.
|
||||
|
||||
Danbooru also has periodic maintenance tasks that run in the background as cron
|
||||
jobs. These are different from the jobs in this directory. See [app/logical/danbooru_maintenance.rb](../logical/danbooru_maintenance.rb).
|
||||
jobs. These are different from the jobs in this directory. See
|
||||
[app/logical/danbooru_maintenance.rb](../logical/danbooru_maintenance.rb).
|
||||
|
||||
# Usage
|
||||
|
||||
Start a pool of job workers:
|
||||
|
||||
```
|
||||
RAILS_ENV=production bin/delayed_job --pool=default:8 --pool=bulk_update start
|
||||
RAILS_ENV=production bin/good_job start --max-threads=4
|
||||
```
|
||||
|
||||
# Examples
|
||||
@@ -47,12 +43,12 @@ DeleteFavoritesJob.perform_later(user)
|
||||
# See also
|
||||
|
||||
* [app/logical/danbooru_maintenance.rb](../logical/danbooru_maintenance.rb)
|
||||
* [app/controllers/delayed_jobs_controller.rb](../controllers/delayed_jobs_controller.rb)
|
||||
* [config/initializers/delayed_jobs.rb](../../config/initializers/delayed_jobs.rb)
|
||||
* [app/controllers/jobs_controller.rb](../controllers/jobs_controller.rb)
|
||||
* [config/initializers/good_job.rb](../../config/initializers/good_job.rb)
|
||||
* [test/jobs](../../test/jobs)
|
||||
|
||||
# External links
|
||||
|
||||
* https://guides.rubyonrails.org/active_job_basics.html
|
||||
* https://github.com/collectiveidea/delayed_job
|
||||
* https://danbooru.donmai.us/delayed_jobs
|
||||
* https://github.com/bensheldon/good_job
|
||||
* https://danbooru.donmai.us/jobs
|
||||
|
||||
@@ -3,15 +3,19 @@
|
||||
# The base class for all background jobs on Danbooru.
|
||||
#
|
||||
# @see https://guides.rubyonrails.org/active_job_basics.html
|
||||
# @see https://github.com/collectiveidea/delayed_job
|
||||
# @see https://github.com/bensheldon/good_job
|
||||
class ApplicationJob < ActiveJob::Base
|
||||
class JobTimeoutError < StandardError; end
|
||||
|
||||
queue_as :default
|
||||
queue_with_priority 0
|
||||
|
||||
around_perform do |_job, block|
|
||||
CurrentUser.scoped(User.system, "127.0.0.1") do
|
||||
ApplicationRecord.without_timeout do
|
||||
block.call
|
||||
Timeout.timeout(8.hours, JobTimeoutError) do
|
||||
block.call
|
||||
end
|
||||
end
|
||||
end
|
||||
end
|
||||
|
||||
@@ -5,13 +5,6 @@
|
||||
#
|
||||
# @see BigqueryExportService
|
||||
class BigqueryExportJob < ApplicationJob
|
||||
retry_on Exception, attempts: 0
|
||||
|
||||
# XXX delayed_job specific
|
||||
def max_attempts
|
||||
1
|
||||
end
|
||||
|
||||
def perform(model:, **options)
|
||||
BigqueryExportService.new(model, **options).export!
|
||||
end
|
||||
|
||||
@@ -3,13 +3,6 @@
|
||||
# A job that sends notifications about new forum posts to Discord. Spawned by
|
||||
# the {ForumPost} class when a new forum post is created.
|
||||
class DiscordNotificationJob < ApplicationJob
|
||||
retry_on Exception, attempts: 0
|
||||
|
||||
# XXX delayed_job specific
|
||||
def max_attempts
|
||||
1
|
||||
end
|
||||
|
||||
def perform(forum_post:)
|
||||
forum_post.send_discord_notification
|
||||
end
|
||||
|
||||
@@ -5,13 +5,6 @@
|
||||
# @see {BulkUpdateRequestProcessor}
|
||||
# @see {BulkUpdateRequest}
|
||||
class ProcessBulkUpdateRequestJob < ApplicationJob
|
||||
retry_on Exception, attempts: 0
|
||||
|
||||
# XXX delayed_job specific
|
||||
def max_attempts
|
||||
1
|
||||
end
|
||||
|
||||
def perform(bulk_update_request)
|
||||
bulk_update_request.processor.process!
|
||||
end
|
||||
|
||||
@@ -1,9 +0,0 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
# A job that runs daily to delete all stale delayed jobs. Spawned by
|
||||
# {DanbooruMaintenance}.
|
||||
class PruneDelayedJobsJob < ApplicationJob
|
||||
def perform
|
||||
Delayed::Job.where("created_at < ?", 45.days.ago).delete_all
|
||||
end
|
||||
end
|
||||
8
app/jobs/prune_jobs_job.rb
Normal file
8
app/jobs/prune_jobs_job.rb
Normal file
@@ -0,0 +1,8 @@
|
||||
# frozen_string_literal: true
|
||||
|
||||
# A job that runs daily to delete all stale jobs. Spawned by {DanbooruMaintenance}.
|
||||
class PruneJobsJob < ApplicationJob
|
||||
def perform
|
||||
GoodJob::ActiveJobJob.where("created_at < ?", 7.days.ago).destroy_all
|
||||
end
|
||||
end
|
||||
@@ -12,7 +12,7 @@ module DanbooruMaintenance
|
||||
end
|
||||
|
||||
def daily
|
||||
queue PruneDelayedJobsJob
|
||||
queue PruneJobsJob
|
||||
queue PrunePostDisapprovalsJob
|
||||
queue PruneBulkUpdateRequestsJob
|
||||
queue PruneBansJob
|
||||
|
||||
Reference in New Issue
Block a user