Make uploads faster by generating and saving thumbnails in parallel.
We generate each thumbnail in parallel, then send each thumbnail to the
backend image servers in parallel.
Most images have 5 variants: 'preview' (150x150), 180x180, 360x360,
720x720, and 'sample' (850px width). Plus the original file, that's 6
files we have to save. In production we have 2 image servers, so we have
to save each file twice, to 2 remote servers. Doing all this in parallel
should make uploads significantly faster.
Remove the SFTP file storage backend. Downstream users can use either
sshfs (which is what Danbooru now uses in production) or rclone instead.
The Ruby SFTP gem was much slower than sshfs.
Move more of the file-handling logic from UploadService and
StorageManager into MediaAsset. This is part of refactoring posts and
uploads to allow multiple images per post.
Remove StorageManager::Hybrid and StorageManager::Match. These were used
to store uploads on different servers based on the post ID or file
sample type. This is no longer used in production because in hindsight
it's a lot more difficult to manage uploads when they're fragmented
across different servers.
If you need this, you can do tricks with network filesystems to get the
same effect. For example, if you want to store some files on server A
and others on server B, then mount servers A and B as network
filesystems (with e.g. sshfs, Samba, NFS, etc), and use symlinks to
point subdirectories at either server A or B.
Fix uploads failing in the default configuration because the backup
storage manager didn't initialize the null storage manager with the
required `base_url` and `base_dir` options.
Fix bug where pruning uploads failed because the rclone binary couldn't
be found. The upload pruner runs under cron, which has a fixed default
$PATH of "/bin:/usr/bin", but in production rclone is installed under
/usr/local/bin. This caused the upload pruner to fail, which prevented
the rest of daily maintenance from running.
* B2 doesn't allow the path to start with a '/' character.
* When storing the file, we have to rewind the file pointer to make sure
we get the whole file.
