jaculabilis/danbooru
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
10,563 Commits 1 Branch 0 Tags
4cc8dd41eca45e79ad107dc554bfdfdddbdce958
Commit Graph

2 Commits

Author SHA1 Message Date
evazion
4b61a9c73c /robots.txt: enable HTTP caching. 
Cache /robots.txt at the HTTP level because it rarely changes but it
gets requested by bots relatively frequently.
 2021-03-07 18:35:37 -06:00
evazion
74d77c1e23 Update robots.txt. 
* Generate /robots.txt dynamically.
* Include link to sitemap.
* Update list of allowed urls.
* Disallow crawling of non-canonical subdomains.
 2020-01-31 02:43:07 -06:00