seo: increase sitemap coverage.

Rework sitemaps to provide more coverage of the site. We want every
important page on the site - including every post, tag, and wiki page -
to be indexed by Google. We do this by generating sitemaps and sitemap
indexes that contain links to every important page on the site.

app/views/robots/index.text.erb

@@ -1,20 +1,19 @@
-Sitemap: <%= root_url %>sitemap.xml
-
 User-agent: *
 Disallow: /
+Allow: /$
 
-<% if Rails.env.production? && Danbooru.config.hostname == request.host %>
+<% if !Rails.env.production? || Danbooru.config.hostname == request.host %>
 Disallow: /*.atom
 Disallow: /*.json
-Disallow: /*.xml
 
-Allow: /$
 Allow: /artists
 Allow: /artist_commentaries
 Allow: /comments
 Allow: /explore
+Allow: /favorite_groups
 Allow: /forum_posts
 Allow: /forum_topics
+Allow: /iqdb_queries
 Allow: /login
 Allow: /notes
 Allow: /pools
@@ -23,7 +22,7 @@ Allow: /sessions
 Allow: /static
 Allow: /tags
 Allow: /uploads
-Allow: /user_upgrades
+Allow: /user_upgrade
 Allow: /users
 Allow: /wiki_pages
 
@@ -51,4 +50,12 @@ Allow: /packs
 Allow: /terms_of_service
 Allow: /privacy
 Allow: /sitemap.xml
+
+Sitemap: <%= sitemap_url(format: :xml, sitemap: "artists") %>
+Sitemap: <%= sitemap_url(format: :xml, sitemap: "forum_topics") %>
+Sitemap: <%= sitemap_url(format: :xml, sitemap: "pools") %>
+Sitemap: <%= sitemap_url(format: :xml, sitemap: "posts") %>
+Sitemap: <%= sitemap_url(format: :xml, sitemap: "tags") %>
+Sitemap: <%= sitemap_url(format: :xml, sitemap: "users") %>
+Sitemap: <%= sitemap_url(format: :xml, sitemap: "wiki_pages") %>
 <% end %>