Search code examples
umbracorobots.txtumbraco7

Robots.txt for umbraco

I wonder if there is a default Robots.txt for Umbraco (7.5) websites. Right now, I created one with these items. Shall I add anything else?

User-Agent: *
Disallow: /umbraco/
Disallow: /umbraco_client/
Disallow: /css
Disallow: /Errorpages/
Disallow: /scripts/
Disallow: /Config/
Solution

  • Regarding CSS & JSS check this: https://yoast.com/dont-block-your-css-and-js-files/. I decided to cut them off lately.

    Mine, always used with Umbraco instances, robots.txt file looks like this:

    # robots.txt for Umbraco
User-agent: *
Disallow: /aspnet_client/
Disallow: /bin/
Disallow: /config/
Disallow: /data/
Disallow: /install/
Disallow: /macroScripts/
Disallow: /masterpages/
Disallow: /umbraco/
Disallow: /umbraco_client/
Disallow: /usercontrols/
Disallow: /xslt/
User-agent: Libwww-perl
Disallow: /
Sitemap: http://{HTTP_HOST}/sitemap