1653392361
Stop updating the fav_string attribute on posts. The column still exists on the table, but is no longer used or updated. Like the pool_string in7d503f08, the fav_string was used in the past to facilitate `fav:X` searches. Posts had a hidden fav_string column that contained a list of every user who favorited the post. These were treated like fake hidden tags on the post so that a search for `fav:X` was treated like a tag search. The fav_string attribute has been unused for search purposes for a while now. It was only kept because of technicalities that required departitioning the favorites table first (340e1008e) before it could be removed. Basically, removing favorites with `@favorite.destroy` was slow because Rails always deletes object by ID, but we didn't have an index on favorites.id, and we couldn't easily add one until the favorites table was departitioned. Fixes #4652. See https://github.com/danbooru/danbooru/issues/4652#issuecomment-754993802 for more discussion of issues caused by the fav_string (in short: write amplification, post table bloat, and favorite inconsistency problems).
Jobs
This directory contains background jobs used by Danbooru. Jobs are used to handle slow-running tasks that need to run in the background, such as processing uploads or bulk update requests. They're also used for asynchronous tasks, such as sending emails, that may temporarily fail but can be automatically retried later.
Jobs use the Rails Active Job framework. Active Job is a common framework that allows jobs to be run on different job runner backends.
In the production environment, jobs are run using the Delayed Job backend. Jobs
are stored in the database in the delayed_job table. Worker processes spawned
by bin/delayed_job poll the table for new jobs to work.
In the development environment, jobs are run with an in-process thread pool. This will run jobs in the background, but will drop jobs when the server is restarted.
There are two job queues, the default queue and the bulk_update. The
bulk_update queue handles bulk update requests. It has only one worker so that
bulk update requests are effectively processed sequentially. The default queue
handles everything else.
There is a very minimal admin dashboard for jobs at https://danbooru.donmai.us/delayed_jobs.
Danbooru also has periodic maintenance tasks that run in the background as cron jobs. These are different from the jobs in this directory. See app/logical/danbooru_maintenance.rb.
Usage
Start a pool of job workers:
RAILS_ENV=production bin/delayed_job --pool=default:8 --pool=bulk_update start
Examples
Spawn a job to be worked in the background. It will be worked as soon as a worker is available:
DeleteFavoritesJob.perform_later(user)
See also
- app/logical/danbooru_maintenance.rb
- app/controllers/delayed_jobs_controller.rb
- config/initializers/delayed_jobs.rb
- test/jobs