629a634b22
/packs was blocked by robots.txt, which prevented Googlebot from fetching CSS when indexing pages, which made Google penalize pages for being mobile unfriendly because it couldn't load the CSS and it thought the layout was broken.
50 lines
768 B
Plaintext
50 lines
768 B
Plaintext
Sitemap: <%= root_url %>sitemap.xml
|
|
|
|
User-agent: *
|
|
Disallow: /
|
|
|
|
<% if Rails.env.production? && Danbooru.config.hostname == request.host %>
|
|
Disallow: /*.atom
|
|
Disallow: /*.json
|
|
Disallow: /*.xml
|
|
|
|
Allow: /$
|
|
Allow: /artists
|
|
Allow: /artist_commentaries
|
|
Allow: /comments
|
|
Allow: /explore
|
|
Allow: /forum_posts
|
|
Allow: /forum_topics
|
|
Allow: /login
|
|
Allow: /notes
|
|
Allow: /pools
|
|
Allow: /posts
|
|
Allow: /sessions
|
|
Allow: /static
|
|
Allow: /tags
|
|
Allow: /uploads
|
|
Allow: /user_upgrades
|
|
Allow: /users
|
|
Allow: /wiki_pages
|
|
|
|
<%# Legacy redirects %>
|
|
Allow: /artist
|
|
Allow: /comment
|
|
Allow: /forum
|
|
Allow: /note
|
|
Allow: /pool
|
|
Allow: /post
|
|
Allow: /tag
|
|
Allow: /user
|
|
Allow: /wiki
|
|
|
|
<%# Images %>
|
|
Allow: /crop
|
|
Allow: /preview
|
|
Allow: /original
|
|
Allow: /sample
|
|
Allow: /data
|
|
Allow: /images
|
|
Allow: /packs
|
|
<% end %>
|