Update robots.txt.

* Generate /robots.txt dynamically.
* Include link to sitemap.
* Update list of allowed urls.
* Disallow crawling of non-canonical subdomains.

app/controllers/robots_controller.rb

@@ -0,0 +1,6 @@
+class RobotsController < ApplicationController
+  respond_to :text
+
+  def index
+  end
+end