91587aeb6b
Block Googlebot from crawling certain slow useless URLs. Sometimes Googlebot tries to crawl old source:<url>, approver:<name>, and ordfav:<name> searches in bulk, which tends to slow down the site because things like source:<url> are inherently slow, and because Google spends hours at a time crawling them in parallel. This is despite the fact that these links are already marked as nofollow and noindex, and source:<url> links were removed from posts a long time ago to try to stop Google from crawling them.
2.0 KiB
2.0 KiB