uploads: fix temp files not being cleaned up quickly enough.

Fix temp files generated during the upload process not being cleaned up quickly enough. This included
downloaded files, generated preview images, and Ugoira video conversions.

Before we relied on `Tempfile` cleaning up files automatically. But this only happened when the
Tempfile object was garbage collected, which could take a long time. In the meantime we could have
hundreds of megabytes of temp files hanging around.

The fix is to explicitly close temp files when we're done with them. But the standard `Tempfile`
class doesn't immediately delete the file when it's closed. So we also have to introduce a
Danbooru::Tempfile wrapper that deletes the tempfile as soon as it's closed.
This commit is contained in:
evazion
2022-11-15 17:34:59 -06:00
parent 21a779455f
commit e935f01358
14 changed files with 67 additions and 27 deletions

View File

@@ -171,11 +171,11 @@ module Danbooru
# Download a file from `url` and return a {MediaFile}.
#
# @param url [String] the URL to download
# @param file [Tempfile] the file to download the URL to
# @param file [Danbooru::Tempfile] the file to download the URL to
# @raise [DownloadError] if the server returns a non-200 OK response
# @raise [FileTooLargeError] if the file exceeds Danbooru's maximum download size.
# @return [Array<(HTTP::Response, MediaFile)>] the HTTP response and the downloaded file
def download_media(url, file: Tempfile.new("danbooru-download-", binmode: true))
def download_media(url, file: Danbooru::Tempfile.new("danbooru-download-#{url.parameterize.truncate(96)}-", binmode: true))
response = get(url)
raise DownloadError, "#{url} failed with code #{response.status}" if response.status != 200

View File

@@ -0,0 +1,14 @@
# frozen_string_literal: true
# Like Tempfile, but delete the tempfile when it's closed.
#
# The Tempfile class in the standard library doesn't delete the file immediately when you call `file.close`. Instead you
# have to call `file.close!` or `file.unlink` to delete the file, or wait until the object gets garbage collected, which
# can take a long time. This makes it so that Tempfiles are cleaned up immediately on close.
module Danbooru
class Tempfile < ::Tempfile
def close(unlink = true)
super
end
end
end