* Mark /comments/:id links in comment timestamps as nofollow to prevent
Googlebot from crawling these links.
* Mark /posts/:id/show_seq links as disallowed in robots.txt to prevent
Googlebot from crawling forward/back links on posts.
Block Googlebot from crawling certain slow useless URLs. Sometimes
Googlebot tries to crawl old source:<url>, approver:<name>, and
ordfav:<name> searches in bulk, which tends to slow down the site because
things like source:<url> are inherently slow, and because Google spends
hours at a time crawling them in parallel. This is despite the fact that
these links are already marked as nofollow and noindex, and source:<url>
links were removed from posts a long time ago to try to stop Google from
crawling them.
Some of these pages were accidentally crawled because rules like `Allow:
/artist` allowed `/artist_versions` to be crawled (Allow rules are
prefix matches).
Rework sitemaps to provide more coverage of the site. We want every
important page on the site - including every post, tag, and wiki page -
to be indexed by Google. We do this by generating sitemaps and sitemap
indexes that contain links to every important page on the site.
/packs was blocked by robots.txt, which prevented Googlebot from
fetching CSS when indexing pages, which made Google penalize pages for
being mobile unfriendly because it couldn't load the CSS and it thought
the layout was broken.
