evazion 74d77c1e23 Update robots.txt. ...

* Generate /robots.txt dynamically.
* Include link to sitemap.
* Update list of allowed urls.
* Disallow crawling of non-canonical subdomains.

2020-01-31 02:43:07 -06:00

..

index.text.erb

Update robots.txt.

2020-01-31 02:43:07 -06:00