4a525c7473
Fix maintenance tasks failing to run in production. In production they were losing the database connection and not re-establishing it, so they couldn't queue jobs. `ApplicationRecord.verify!` will check if the connection is lost and re-establish it if it is. The database connection was being lost because in production we use a Kubernetes service IP for the database IP, which is essentially a virtual IP that maps to the real IP. This mapping is implemented with IPVS[1][2], which has a default idle connection timeout of 5 minutes. If the connection isn't used for more than 5 minutes, then it's closed. Since maintenance only runs once an hour, the database connection would be lost because it was idle for too long. 1: https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-ipvs 2: https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/
39 lines
820 B
Ruby
39 lines
820 B
Ruby
module DanbooruMaintenance
|
|
module_function
|
|
|
|
def hourly
|
|
queue PruneUploadsJob
|
|
queue PrunePostsJob
|
|
queue PruneRateLimitsJob
|
|
queue RegeneratePostCountsJob
|
|
end
|
|
|
|
def daily
|
|
queue PruneDelayedJobsJob
|
|
queue PrunePostDisapprovalsJob
|
|
queue PruneBulkUpdateRequestsJob
|
|
queue PruneBansJob
|
|
queue BigqueryExportAllJob
|
|
queue VacuumDatabaseJob
|
|
end
|
|
|
|
def weekly
|
|
queue RetireTagRelationshipsJob
|
|
queue DmailInactiveApproversJob
|
|
end
|
|
|
|
def monthly
|
|
queue PruneApproversJob
|
|
end
|
|
|
|
def queue(job)
|
|
Rails.logger.level = :info
|
|
DanbooruLogger.info("Queueing #{job.name}")
|
|
ApplicationRecord.connection.verify!
|
|
job.perform_later
|
|
rescue Exception => e # rubocop:disable Lint/RescueException
|
|
DanbooruLogger.log(exception)
|
|
raise e if Rails.env.test?
|
|
end
|
|
end
|