evazion
|
a7dc05ce63
|
Enable frozen string literals.
Make all string literals immutable by default.
|
2021-12-14 21:33:27 -06:00 |
|
evazion
|
4b61a9c73c
|
/robots.txt: enable HTTP caching.
Cache /robots.txt at the HTTP level because it rarely changes but it
gets requested by bots relatively frequently.
|
2021-03-07 18:35:37 -06:00 |
|
evazion
|
74d77c1e23
|
Update robots.txt.
* Generate /robots.txt dynamically.
* Include link to sitemap.
* Update list of allowed urls.
* Disallow crawling of non-canonical subdomains.
|
2020-01-31 02:43:07 -06:00 |
|