docs: document config/ and some directories in app/.

* Add README files to several directories in app/ giving a brief
  overview of some parts of Danbooru's architecture.
* Add documentation for files in config/.

app/components/README.md

@@ -0,0 +1,21 @@
+# Components
+
+Components are used to encapsulate common UI widgets used throughout the site. This includes things like comments, tag
+lists, post thumbnails, paginators, and other things used repeatedly throughout the site. Components encapsulate a
+widget's HTML, CSS, and Javascript together so that all the code for a component is in a single place.
+
+This uses the [ViewComponent](https://github.com/github/view_component) library.
+
+Components are similar in concept to React components.
+
+# See also
+
+* [app/views](../views)
+* [app/javascript](../javascript)
+* [test/components](../../test/components)
+
+# External links
+
+* https://viewcomponent.org
+* https://github.com/github/view_component
+* https://github.blog/2020-12-15-encapsulating-ruby-on-rails-views/

app/controllers/README.md

@@ -0,0 +1,164 @@
+# Controllers
+
+Controllers are the entry points to Danbooru. Every URL on the site corresponds to a controller action. When a request
+for an URL is made, the corresponding controller action is called to handle the request.
+
+Controllers follow a convention where, for example, the URL <https://danbooru.donmai.us/posts/1234> is handled by the
+`#show` method inside the `PostsController` living at [app/controllers/posts_controller.rb](posts_controller.rb).
+<https://danbooru.donmai.us/posts?tags=touhou> is handled by the `#index` method in the PostsController. The HTML
+template for the response lives at [app/views/posts/index.html.erb](../views/posts/index.html.erb). See below for more
+examples.
+
+Controllers are responsible for taking the URL parameters, checking whether the user is authorized to perform the
+action, actually performing the action, then returning the response. Most controllers simply fetch or update a model,
+then render an HTML template from [app/views](../views) in response.
+
+# Example
+
+A standard controller looks something like this:
+
+```ruby
+class BansController < ApplicationController
+  def new
+    @ban = authorize Ban.new(permitted_attributes(Ban))
+    respond_with(@ban)
+  end
+
+  def edit
+    @ban = authorize Ban.find(params[:id])
+    respond_with(@ban)
+  end
+
+  def index
+    @bans = authorize Ban.paginated_search(params, count_pages: true)
+    respond_with(@bans)
+  end
+
+  def show
+    @ban = authorize Ban.find(params[:id])
+    respond_with(@ban)
+  end
+
+  def create
+    @ban = authorize Ban.new(banner: CurrentUser.user, **permitted_attributes(Ban))
+    @ban.save
+    respond_with(@ban, location: bans_path)
+  end
+
+  def update
+    @ban = authorize Ban.find(params[:id])
+    @ban.update(permitted_attributes(@ban))
+    respond_with(@ban)
+  end
+
+  def destroy
+    @ban = authorize Ban.find(params[:id])
+    @ban.destroy
+    respond_with(@ban)
+  end
+end
+```
+
+# Routes
+
+Each controller action above corresponds to an URL:
+
+| Controller Action      | URL                                         | Route               | Route Helper        | View                          |
+|------------------------|---------------------------------------------|---------------------|---------------------|-------------------------------|
+| BansController#new     | https://danbooru.donmai.us/bans/new         | GET /bans/new       | new_ban_path        | app/views/bans/new.html.erb   |
+| BansController#edit    | https://danbooru.donmai.us/bans/1234/edit   | GET /bans/:id/edit  | edit_ban_path(@ban) | app/views/bans/edit.html.erb  |
+| BansController#index   | https://danbooru.donmai.us/bans             | GET /bans           | bans_path           | app/views/bans/index.html.erb |
+| BansController#show    | https://danbooru.donmai.us/bans/1234        | GET /bans/:id       | ban_path(@ban)      | app/views/bans/show.html.erb  |
+| BansController#create  | POST https://danbooru.donmai.us/bans        | POST /bans          |                     |                               |
+| BansController#update  | PUT https://danbooru.donmai.us/bans/1234    | PUT /bans/:id       |                     |                               |
+| BansController#destroy | DELETE https://danbooru.donmai.us/bans/1234 | DELETE /bans/:id    |                     |                               |
+
+These routes are defined in [config/routes.rb](../config/routes.rb).
+
+# Authorization
+
+Most permission checks for whether a user has permission to do something happen inside controllers, using `authorize`
+calls.
+
+The `authorize` method comes from the [Pundit](https://github.com/varvet/pundit) framework. This method checks whether
+the current user is authorized to perform the current action. If not, it raises a `Pundit::NotAuthorizedError`, which is
+caught in the `ApplicationController`.
+
+The actual authorization logic for these calls lives in [app/policies](../policies). They follow a convention where the
+authorization logic for the `BansController#create` action lives in `BanPolicy#create?`, which lives in
+[app/policies/ban_policy.rb](../policies/ban_policy.rb). The call to `authorize` in the controller simply finds and
+calls the ban policy.
+
+The `#create`, `#new`, and `#update` actions also use `permitted_attributes` to check that the user is allowed to update
+the model attributes they're trying to update. This also comes from the Pundit framework. See the
+`permitted_attributes_for_create` and `permitted_attributes_for_update` methods in [app/policies](../policies).
+
+# Responses
+
+Controllers use `respond_with(@post)` to generate a response. This comes from the [Responders](https://github.com/heartcombo/responders)
+gem. `respond_with` does the following:
+
+* Detects whether the user wants an HTML, JSON, or XML response.
+* Renders an HTML template from [app/views](../views) for HTML requests.
+* Renders JSON or XML for API requests.
+* Handles universal URL parameters, like `only` or `includes`.
+* Handles universal behavior, like returning 200 OK for successful responses, or returning an error if trying
+  to save a model with validation errors.
+
+# HTML Responses
+
+The HTML templates for controller actions live in [app/views](../views). For example, the template for
+PostsController#show, which corresponds to https://danbooru.donmai.us/posts/1234, lives in
+[app/views/posts/show.html.erb](../views/posts/show.html.erb).
+
+Instance variables set by controllers are automatically inherited by views. For example, if a controller sets `@post`,
+then the `@post` variable will be available in the view.
+
+# API Responses
+
+All URLs support JSON or XML responses. This is handled by `respond_with`.
+
+The response format can be chosen in several ways. First, by adding a .json or .xml file extension:
+
+* https://danbooru.donmai.us/posts.json
+* https://danbooru.donmai.us/posts.xml
+
+Second, by setting the `format` URL parameter:
+
+* https://danbooru.donmai.us/posts?format=json
+* https://danbooru.donmai.us/posts?format=xml
+
+Third, by setting the `Accept` HTTP header:
+
+```sh
+curl -H "Accept: application/json" https://danbooru.donmai.us/posts
+curl -H "Accept: application/xml" https://danbooru.donmai.us/posts
+```
+
+When generating API responses, `respond_with` uses the `api_attributes` method inside [app/policies](../policies) to
+determine which attributes are visible to the current user.
+
+# Application Controller
+
+Global behavior that runs on every request lives inside the [ApplicationController](application_controller.rb). This
+includes the following:
+
+* Setting the current user based on their session cookies or API keys.
+* Checking rate limits.
+* Checking for IP bans.
+* Adding certain HTTP headers.
+* Handling exceptions.
+
+# See also
+
+* [app/models](../models)
+* [app/policies](../policies)
+* [app/views](../views)
+* [config/routes.rb](../../config/routes.rb)
+* [test/functional](../../test/functional)
+
+# External links
+
+* https://guides.rubyonrails.org/action_controller_overview.html
+* https://github.com/heartcombo/responders
+* https://github.com/varvet/pundit

app/helpers/README.md

@@ -0,0 +1,25 @@
+# Helpers
+
+This directory contains helper functions used by views. Helpers are used for simple common functions, such as linking to
+users or formatting timestamps.
+
+Helper functions are globals. If you see an unnamespaced function in a view, and it's not a Rails function, then it's
+probably a helper defined here.
+
+All helper functions defined in this directory are globally available to all views. They're not limited to single views.
+For example, the functions in [posts_helper.rb](posts_helper.rb) are available to all views, not just to
+[app/views/posts](../views/posts).
+
+The use of helper functions should be minimized. Partials or components are preferred for more complex widgets, or for
+things used in only one or two places. Helper functions should be limited to very simple things used in nearly all
+views.
+
+# See also
+
+* [app/views](../views)
+* [app/components](../components)
+
+# External links
+
+* https://api.rubyonrails.org/classes/ActionController/Helpers.html
+* https://www.rubyguides.com/2020/01/rails-helpers/

app/jobs/README.md

@@ -0,0 +1,58 @@
+# Jobs
+
+This directory contains background jobs used by Danbooru. Jobs are used to
+handle slow-running tasks that need to run in the background, such as processing
+uploads or bulk update requests. They're also used for asynchronous tasks, such
+as sending emails, that may temporarily fail but can be automatically retried
+later.
+
+Jobs use the Rails Active Job framework. Active Job is a common framework that
+allows jobs to be run on different job runner backends.
+
+In the production environment, jobs are run using the Delayed Job backend.  Jobs
+are stored in the database in the `delayed_job` table. Worker processes spawned
+by `bin/delayed_job` poll the table for new jobs to work.
+
+In the development environment, jobs are run with an in-process thread pool.
+This will run jobs in the background, but will drop jobs when the server is
+restarted.
+
+There are two job queues, the `default` queue and the `bulk_update`. The
+`bulk_update` queue handles bulk update requests. It has only one worker so that
+bulk update requests are effectively processed sequentially. The `default` queue
+handles everything else.
+
+There is a very minimal admin dashboard for jobs at https://danbooru.donmai.us/delayed_jobs.
+
+Danbooru also has periodic maintenance tasks that run in the background as cron
+jobs. These are different from the jobs in this directory. See [app/logical/danbooru_maintenance.rb](../logical/danbooru_maintenance.rb).
+
+# Usage
+
+Start a pool of job workers:
+
+```
+RAILS_ENV=production bin/delayed_job --pool=default:8 --pool=bulk_update start
+```
+
+# Examples
+
+Spawn a job to be worked in the background. It will be worked as soon as a
+worker is available:
+
+```ruby
+DeleteFavoritesJob.perform_later(user)
+```
+
+# See also
+
+* [app/logical/danbooru_maintenance.rb](../logical/danbooru_maintenance.rb)
+* [app/controllers/delayed_jobs_controller.rb](../controllers/delayed_jobs_controller.rb)
+* [config/initializers/delayed_jobs.rb](../../config/initializers/delayed_jobs.rb)
+* [test/jobs](../../test/jobs)
+
+# External links
+
+* https://guides.rubyonrails.org/active_job_basics.html
+* https://github.com/collectiveidea/delayed_job
+* https://danbooru.donmai.us/delayed_jobs

app/logical/README.md

@@ -0,0 +1,12 @@
+# Logical
+
+This directory contains library code used through Danbooru. This includes things like defining API clients, dealing with
+sources, parsing tag searches, storing and resizing images, and so on.
+
+Many of the files here use the Service Object pattern. Instead of putting complex code in models or controllers, it goes
+here, in plain old Ruby objects (POROs). This keeps models and controllers simpler, and keeps domain logic isolated and
+independent from the database and the HTTP request cycle.
+
+# External links
+
+* https://www.codewithjason.com/rails-service-objects/

app/mailers/README.md

@@ -0,0 +1,31 @@
+# Mailers
+
+This directory contains mailers for sending emails. Mailers are kind of like controllers, except for generating emails
+instead of generating HTML.
+
+The actual email templates live in [app/views/user_mailer](../views/user_mailer).
+
+Emails are sent asynchronously using a background job. If sending the email fails, it will be retried later.
+
+Sending emails requires a SMTP server to be configured in
+[config/danbooru_local_config.rb](../../config/danbooru_local_config.rb). In production,
+[Amazon SES](https://aws.amazon.com/ses/) is used to send emails.
+
+Email templates can be previewed at http://localhost:3000/rails/mailers (assuming you're running `bin/rails server` on
+port 3000, the default).
+
+# Example
+
+```ruby
+UserMailer.welcome_user(@user).deliver_later
+```
+
+# See also
+
+* [app/views/user_mailer](../views/user_mailer)
+* [test/mailers/previews/user_mailer_preview.rb](../../test/mailers/previews/user_mailer_preview.rb)
+
+# External links
+
+* https://guides.rubyonrails.org/action_mailer_basics.html
+* https://guides.rubyonrails.org/testing.html#testing-your-mailers

bin/README.md

@@ -0,0 +1,4 @@
+# Bin
+
+This directory contains executable scripts. Most of the scripts here come from Ruby gems installed by `bundle install`
+and shouldn't be edited by end users.

config/README.md

@@ -0,0 +1,14 @@
+# Config
+
+This directory contains configuration files for Danbooru.
+
+To configure your Danbooru instance, copy [danbooru_default_config.rb](./danbooru_default_config.rb)
+to [danbooru_local_config.rb](./danbooru_local_config.rb) and edit it. See
+[danbooru_default_config.rb](./danbooru_default_config.rb) for details.
+
+The only file here that end users need to be concerned about is [danbooru_default_config.rb](./danbooru_default_config.rb).
+The rest of the files here are internal Rails-related configuration files that end users shouldn't need to edit.
+
+# External links
+
+* https://guides.rubyonrails.org/configuring.html

config/application.rb

@@ -1,4 +1,10 @@
+# This file runs after config/boot.rb and before config/environment.rb. It loads Rails, loads the gems, loads the
+# Danbooru configuration, and does some basic Rails configuration.
+#
+# @see https://guides.rubyonrails.org/initialization.html
+
 require_relative 'boot'
+
 require "rails"
 require "active_record/railtie"
 # require "active_storage/engine"
@@ -12,8 +18,11 @@ require "active_job/railtie"
 require "rails/test_unit/railtie"
 # require "sprockets/railtie"
 
+# Load the gems for the current Rails environment from the Gemfile.
 Bundler.require(*Rails.groups)
 
+# Load the default Danbooru configuration from config/danbooru_default_config.rb and the custom config from
+# config/danbooru_local_config.rb.
 begin
   require_relative "danbooru_default_config"
   require_relative ENV.fetch("DANBOORU_CONFIG_FILE", "danbooru_local_config")

config/boot.rb

@@ -1,4 +1,14 @@
+# This is the first file that runs during Rails boot. The next files to run are
+# config/application.rb, then config/environment.rb.
+#
+# @see https://guides.rubyonrails.org/initialization.html
+# @see https://github.com/Shopify/bootsnap
+
 ENV['BUNDLE_GEMFILE'] ||= File.expand_path('../Gemfile', __dir__)
 
-require "bundler/setup" # Set up gems listed in the Gemfile.
+# Set up the load path for gems listed in the Gemfile. Does not actually load the gems; that happens in
-require "bootsnap/setup" # Speed up boot time by caching expensive operations.
+# config/application.rb.
+require "bundler/setup"
+
+# Speed up boot time by caching expensive operations.
+require "bootsnap/setup"

config/danbooru_default_config.rb

@@ -1,3 +1,26 @@
+# This file contains all the configuration settings for Danbooru.
+#
+# Don't edit this file. Instead, to configure your Danbooru instance, copy this
+# file to config/danbooru_local_config.rb and edit that. Remove all settings you
+# don't need to change, and edit only the settings you do need to change.
+#
+# You can also use environment variables to change settings on the command line.
+# For example, to change the site name, you could do:
+#
+#     DANBOORU_APP_NAME=MyBooru bin/rails server
+#
+# This works with nearly any setting. Just take the setting name, uppercase it,
+# and add DANBOORU_ to the front. More examples:
+#
+#     DANBOORU_CANONICAL_URL=https://booru.example.com
+#     DANBOORU_CONTACT_EMAIL=admin@borou.example.com
+#     DANBOORU_DISCORD_SERVER_URL=https://discord.gg/yourbooru
+#
+# Settings from environment variables will override those from the config file.
+#
+# You can also set these environment variables in an envfile instead of the
+# command line. See the .env file in the root project directory for details.
+#
 module Danbooru
   class Configuration
     # A secret key used to encrypt session cookies, among other things.

config/database.yml

@@ -1,9 +1,28 @@
-# Don't edit this file. To override this file, set `DATABASE_URL` in .env.local
+# This file contains database configuration. Don't edit this file. To override
-# instead. Example:
+# this file, set `DATABASE_URL` in .env.local instead.
 #
-#   DATABASE_URL=postgresql://danbooru:password@localhost/danbooru2
+# By default, with no configuration, we try to connect to the Postgres server
+# running on localhost with username `danbooru` and database name `danbooru2`.
 #
-# https://guides.rubyonrails.org/configuring.html#configuring-a-database
+# Example:
+#
+#   # Connect to the database named `danbooru2` via localhost on port 5432
+#   DATABASE_URL=postgresql://localhost/danbooru2
+#
+#   # Connect via Unix domain socket in /var/run/postgresql
+#   # https://zaiste.net/posts/postgresql-unix-socket-tcpip-loopback/
+#   DATABASE_URL=postgresql://%2Fvar%2Frun%2Fpostgresql/danbooru2
+#
+# The general form for a database URL is:
+#
+#   postgresql://[user[:password]@][host][:port][/dbname][?param1=value1&...]`
+#
+# The default is:
+#
+#   postgresql://danbooru@localhost/danbooru2
+#
+# @see https://guides.rubyonrails.org/configuring.html#configuring-a-database
+# @see https://www.postgresql.org/docs/current/libpq-connect.html#id-1.7.3.8.3.6
 
 default: &default
   adapter: postgresql

config/deploy.rb

@@ -1,3 +1,12 @@
+# This file contains configuration settings for deploying Danbooru to the
+# production servers using Capistrano. This is only used by production and
+# shouldn't be edited by end users.
+#
+# @see Capfile
+# @see config/deploy
+# @see lib/capistrano/tasks
+# @see https://capistranorb.com
+
 set :stages, %w(production development staging)
 set :default_stage, "staging"
 set :application, "danbooru"

config/deploy/README.md

@@ -0,0 +1,14 @@
+# Deploy
+
+This directory contains configuration settings for deploying Danbooru to the production servers
+using Capistrano. This is only used by production and shouldn't be edited by end users.
+
+# See also
+
+* [Capfile](../../Capfile)
+* [config/deploy.rb](../deploy.rb)
+* [lib/capistrano/tasks](../../lib/capistrano/tasks)
+
+# External links
+
+* https://capistranorb.com

config/docker/README.md

@@ -0,0 +1,10 @@
+# Docker
+
+This directory contains files for building Docker images for Danbooru.
+
+# External links
+
+* https://docs.docker.com/get-started/
+* https://docs.docker.com/engine/reference/commandline/cli/
+* https://docs.docker.com/engine/reference/builder/
+* https://hub.docker.com/r/evazion/danbooru

config/environments/README.md

@@ -0,0 +1,15 @@
+# Environments
+
+This directory contains environment-specific Rails configuration. This lets us have different
+configuration settings in development, test, and production modes.
+
+Universal Rails configuration goes in [config/application.rb](../application.rb) or in
+[config/initializers](../initializers).
+
+# See also
+
+* [config/application.rb](../application.rb)
+
+# External links
+
+* https://guides.rubyonrails.org/configuring.html#creating-rails-environments

config/initializers/README.md

@@ -0,0 +1,15 @@
+# Initializers
+
+This directory contains initializers run by Rails during bootup. Initializers are generally used to
+configure libraries or other global settings during application boot.
+
+# See also
+
+* [config/boot.rb](../boot.rb)
+* [config/application.rb](../application.rb)
+* [config/environment.rb](../environment.rb)
+
+# External links
+
+* https://guides.rubyonrails.org/configuring.html#using-initializer-files
+* https://guides.rubyonrails.org/initialization.html

config/puma.rb

@@ -1,3 +1,10 @@
+# This file contains configuration settings for the Puma web server. These
+# settings apply when running Danbooru with `bin/rails server`. In production,
+# Danbooru currently uses Unicorn instead of Puma.
+#
+# @see https://puma.io
+# @see https://github.com/puma/puma
+
 # Puma can serve each request in a thread from an internal thread pool.
 # The `threads` method setting takes two numbers: a minimum and maximum.
 # Any libraries that use thread pools should be configured to match

config/routes.rb

@@ -1,3 +1,12 @@
+# This file contains configuration for Danbooru's URL routes. It defines all the
+# URL endpoints and HTTP redirects used by Danbooru.
+#
+# A list of routes can be found at http://localhost:3000/rails/info/routes when
+# running the server in development mode. You can also run `bin/rails routes` to
+# produce a list of routes.
+#
+# @see https://guides.rubyonrails.org/routing.html
+# @see http://localhost:3000/rails/info/routes
 Rails.application.routes.draw do
   resources :posts, only: [:index, :show, :update, :destroy] do
     get :random, on: :collection

config/schedule.rb

@@ -1,3 +1,9 @@
+# This file is used by the `whenver` gem to generate a crontab that runs
+# Danbooru's maintenance tasks.
+#
+# @see app/logical/danbooru_maintenance.rb
+# @see https://github.com/javan/whenever
+
 # this is used in config/environments/production.rb.
 env "RAILS_LOG_TO_STDOUT", "true"
 

config/solargraph.rb

@@ -1,5 +1,9 @@
-# https://solargraph.org/guides/rails
+# This file is used by the Solargraph language server to add better knowledge of
-# https://gist.github.com/castwide/28b349566a223dfb439a337aea29713e
+# Rails for intellisense purposes for editors like VS Code or Vim that have Ruby
+# language server support configured.
+#
+# @see https://solargraph.org/guides/rails
+# @see https://gist.github.com/castwide/28b349566a223dfb439a337aea29713e
 #
 # The following comments fill some of the gaps in Solargraph's understanding of
 # Rails apps. Since they're all in YARD, they get mapped in Solargraph but

config/spring.rb

@@ -1,3 +1,7 @@
+# This file configures Spring, the Rails application preloader.
+#
+# @see https://github.com/rails/spring
+
 %w[
   .ruby-version
   .rbenv-vars

config/unicorn/README.md

@@ -0,0 +1,12 @@
+# Unicorn
+
+This directory contains configuration files for Unicorn, the web server that Danbooru uses in production.
+
+# See also
+
+* [config/puma.rb](../puma.rb)
+
+# External links
+
+* https://yhbt.net/unicorn/README.html
+* https://github.com/defunkt/unicorn

config/webpack/README.md

@@ -0,0 +1,16 @@
+# Webpack
+
+This directory contains configuration for Webpack. Webpack is used to bundle Javascript and CSS
+assets in production. In Rails, Webpack is wrapped by Webpacker, a layer that tries to provide
+sensible defaults for Webpack.
+
+# See also
+
+* [app/javascript](../../app/javascript)
+* [config/webpacker.yml](../webpacker.yml)
+
+# External links
+
+* https://guides.rubyonrails.org/webpacker.html
+* https://github.com/rails/webpacker#webpack-configuration
+* https://webpack.js.org/configuration/

config/webpack/base.js

@@ -13,6 +13,7 @@ module.exports = merge(webpackConfig, {
   },
   module: {
     rules: [{
+      // https://github.com/usabilityhub/rails-erb-loader
       test: /.erb$/,
       enforce: "pre",
       exclude: /node_modules/,

config/webpack/development.js

@@ -1,3 +1,5 @@
+// @see https://github.com/webpack-contrib/eslint-webpack-plugin
+
 process.env.NODE_ENV = process.env.NODE_ENV || 'development'
 
 const { merge } = require('@rails/webpacker')

config/webpacker.yml

@@ -1,4 +1,10 @@
+# This file configures Webpacker, the Rails wrapper around Webpack. Webpack is
+# used to bundle Javascript and CSS assets.
+#
 # Note: You must restart bin/webpack-dev-server for changes to take effect
+#
+# @see https://github.com/rails/webpacker
+# @see https://edgeguides.rubyonrails.org/webpacker.html
 
 default: &default
   source_path: app/javascript

script/fixes/README.md

@@ -0,0 +1,6 @@
+# Fixes
+
+This directory contains fix scripts designed to fix various problems with the production database. Most of these scripts
+are for the production database and aren't meant to be run by downstream users.
+
+Most of the older scripts here no longer work because of changes to the code over the years.