Commit Graph

7 Commits

Author SHA1 Message Date
evazion
097220fd88 robots.txt: allow /favicon.ico. 2021-01-13 21:26:34 -06:00
evazion
48676789f0 robots.txt: fix hardcoded paths. 2021-01-02 04:09:43 -06:00
evazion
42f0112c38 seo: increase sitemap coverage.
Rework sitemaps to provide more coverage of the site. We want every
important page on the site - including every post, tag, and wiki page -
to be indexed by Google. We do this by generating sitemaps and sitemap
indexes that contain links to every important page on the site.
2020-07-10 00:18:30 -05:00
evazion
d88a2a674f seo: allow /sitemap.xml in robots.txt 2020-07-08 16:16:46 -05:00
evazion
629a634b22 seo: don't block css in robots.txt.
/packs was blocked by robots.txt, which prevented Googlebot from
fetching CSS when indexing pages, which made Google penalize pages for
being mobile unfriendly because it couldn't load the CSS and it thought
the layout was broken.
2020-07-03 15:58:59 -05:00
evazion
e2ec015603 seo: index frontpage, images, and legacy redirects. 2020-06-27 18:09:02 -05:00
evazion
74d77c1e23 Update robots.txt.
* Generate /robots.txt dynamically.
* Include link to sitemap.
* Update list of allowed urls.
* Disallow crawling of non-canonical subdomains.
2020-01-31 02:43:07 -06:00