cron: fix maintenance tasks failing to run.

Fix maintenance tasks failing to run in production. In production they
were losing the database connection and not re-establishing it, so they
couldn't queue jobs. `ApplicationRecord.verify!` will check if the
connection is lost and re-establish it if it is.

The database connection was being lost because in production we use a
Kubernetes service IP for the database IP, which is essentially a
virtual IP that maps to the real IP. This mapping is implemented with
IPVS[1][2], which has a default idle connection timeout of 5 minutes. If
the connection isn't used for more than 5 minutes, then it's closed.
Since maintenance only runs once an hour, the database connection would
be lost because it was idle for too long.

1: https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-ipvs
2: https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/
This commit is contained in:
evazion
2021-09-28 16:36:12 -05:00
parent a1d4408c29
commit 4a525c7473

View File

@@ -29,6 +29,7 @@ module DanbooruMaintenance
def queue(job)
Rails.logger.level = :info
DanbooruLogger.info("Queueing #{job.name}")
ApplicationRecord.connection.verify!
job.perform_later
rescue Exception => e # rubocop:disable Lint/RescueException
DanbooruLogger.log(exception)