4 Commits

Author	SHA1	Message	Date
evazion	d88a2a674f	seo: allow /sitemap.xml in robots.txt	2020-07-08 16:16:46 -05:00
evazion	629a634b22	seo: don't block css in robots.txt. ... /packs was blocked by robots.txt, which prevented Googlebot from fetching CSS when indexing pages, which made Google penalize pages for being mobile unfriendly because it couldn't load the CSS and it thought the layout was broken.	2020-07-03 15:58:59 -05:00
evazion	e2ec015603	seo: index frontpage, images, and legacy redirects.	2020-06-27 18:09:02 -05:00
evazion	74d77c1e23	Update robots.txt. ... * Generate /robots.txt dynamically. * Include link to sitemap. * Update list of allowed urls. * Disallow crawling of non-canonical subdomains.	2020-01-31 02:43:07 -06:00