abdab7a0a8
Rework the upload process so that files are saved to Danbooru first before the user starts tagging the upload. The main user-visible change is that you have to select the file first before you can start tagging it. Saving the file first lets us fix a number of problems: * We can check for dupes before the user tags the upload. * We can perform dupe checks and show preview images for users not using the bookmarklet. * We can show preview images without having to proxy images through Danbooru. * We can show previews of videos and ugoira files. * We can reliably show the filesize and resolution of the image. * We can let the user save files to upload later. * We can get rid of a lot of spaghetti code related to preprocessing uploads. This was the cause of most weird "md5 confirmation doesn't match md5" errors. (Not all of these are implemented yet.) Internally, uploading is now a two-step process: first we create an upload object, then we create a post from the upload. This is how it works: * The user goes to /uploads/new and chooses a file or pastes an URL into the file upload component. * The file upload component calls `POST /uploads` to create an upload. * `POST /uploads` immediately returns a new upload object in the `pending` state. * Danbooru starts processing the upload in a background job (downloading, resizing, and transferring the image to the image servers). * The file upload component polls `/uploads/$id.json`, checking the upload `status` until it returns `completed` or `error`. * When the upload status is `completed`, the user is redirected to /uploads/$id. * On the /uploads/$id page, the user can tag the upload and submit it. * The upload form calls `POST /posts` to create a new post from the upload. * The user is redirected to the new post. This is the data model: * An upload represents a set of files uploaded to Danbooru by a user. Uploaded files don't have to belong to a post. An upload has an uploader, a status (pending, processing, completed, or error), a source (unless uploading from a file), and a list of media assets (image or video files). * There is a has-and-belongs-to-many relationship between uploads and media assets. An upload can have many media assets, and a media asset can belong to multiple uploads. Uploads are joined to media assets through a upload_media_assets table. An upload could potentially have multiple media assets if it's a Pixiv or Twitter gallery. This is not yet implemented (at the moment all uploads have one media asset). A media asset can belong to multiple uploads if multiple people try to upload the same file, or if the same user tries to upload the same file more than once. New features: * On the upload page, you can press Ctrl+V to paste an URL and immediately upload it. * You can save files for upload later. Your saved files are at /uploads. Fixes: * Improved error messages when uploading invalid files, bad URLs, and when forgetting the rating.
Quickstart
Run this to start a basic Danbooru instance:
curl -sSL https://raw.githubusercontent.com/danbooru/danbooru/master/bin/danbooru | sh
This will install Docker Compose and use it to start Danbooru. When it's done, Danbooru will be running at http://localhost:3000.
Alternatively, if you already have Docker Compose installed, you can just do:
wget https://raw.githubusercontent.com/danbooru/danbooru/master/docker-compose.yaml
docker-compose up
Manual Installation
Follow the INSTALL.debian script to install Danbooru.
The INSTALL.debian script is written for Debian, but can be adapted for other distributions. Danbooru has been successfully installed on Debian, Ubuntu, Fedora, Arch, and OS X. It is recommended that you use an Ubuntu-based system since Ubuntu is what is used in development and production.
See here for a guide on how set up Danbooru inside a virtual machine.
For best performance, you will need at least 256MB of RAM for PostgreSQL and Rails. The memory requirement will grow as your database gets bigger.
In production, Danbooru uses PostgreSQL 10.18, but any release later than this should work.
Troubleshooting
If your setup is not working, here are the steps I usually recommend to people:
-
Test the database. Make sure you can connect to it using
psql. Make sure the tables exist. If this fails, you need to work on correctly installing PostgreSQL, importing the initial schema, and running the migrations. -
Test the Rails database connection by using
bin/rails console. RunPost.countto make sure Rails can connect to the database. If this fails, you need to make sure your Danbooru configuration files are correct. -
Test Nginx to make sure it's working correctly. You may need to debug your Nginx configuration file.
-
Check all log files.
Services
Danboou depends on a couple of cloud services and several microservices to implement certain features.
Amazon Web Services
The following features require an Amazon AWS account:
- Pool history
- Post history
Google APIs
The following features require a Google Cloud account:
- BigQuery database export
IQDB Service
IQDB integration is delegated to the IQDB service.
Archive Service
In order to access pool and post histories you will need to install and configure the Archives service.
Reportbooru Service
The following features are delegated to the Reportbooru service:
- Post views
- Missed searches report
- Popular searches report
Recommender Service
Post recommendations require the Recommender service.