From 4a525c747389e3876c176d310275a4a4339b1a40 Mon Sep 17 00:00:00 2001 From: evazion Date: Tue, 28 Sep 2021 16:36:12 -0500 Subject: [PATCH] cron: fix maintenance tasks failing to run. Fix maintenance tasks failing to run in production. In production they were losing the database connection and not re-establishing it, so they couldn't queue jobs. `ApplicationRecord.verify!` will check if the connection is lost and re-establish it if it is. The database connection was being lost because in production we use a Kubernetes service IP for the database IP, which is essentially a virtual IP that maps to the real IP. This mapping is implemented with IPVS[1][2], which has a default idle connection timeout of 5 minutes. If the connection isn't used for more than 5 minutes, then it's closed. Since maintenance only runs once an hour, the database connection would be lost because it was idle for too long. 1: https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-ipvs 2: https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/ --- app/logical/danbooru_maintenance.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/app/logical/danbooru_maintenance.rb b/app/logical/danbooru_maintenance.rb index 4534230d5..be3122ade 100644 --- a/app/logical/danbooru_maintenance.rb +++ b/app/logical/danbooru_maintenance.rb @@ -29,6 +29,7 @@ module DanbooruMaintenance def queue(job) Rails.logger.level = :info DanbooruLogger.info("Queueing #{job.name}") + ApplicationRecord.connection.verify! job.perform_later rescue Exception => e # rubocop:disable Lint/RescueException DanbooruLogger.log(exception)