more crawler constraints

This commit is contained in:
r888888888
2016-02-05 14:14:27 -08:00
parent f2e651a801
commit e04a62b962
3 changed files with 4 additions and 3 deletions

View File

@@ -12,6 +12,7 @@ Disallow: /counts
Disallow: /delayed_jobs
Disallow: /dmails
Disallow: /favorite
Disallow: /favorites
Disallow: /iqdb_queries
Disallow: /ip_bans
Disallow: /janitor_trials