Is it possible to add SEO tags such as 'noodp' to a robots.txt file instead of using <meta> tags? I am trying to avoid messing with our CMS template, although I suspect that I may have to...
Could I try something similar to this...
User-Agent: *
Disallow: /hidden
Sitemap: www.example.com
noodp:
I think robots.txt takes precedence over meta tags? For noindex for instance, the crawler will not even see the page in question. For something like noodp however, is this still the case?
You can't do this with robots.txt, but you can get the same effect using the X-Robots-Tag response header.
Add something like this to the appropriate part of your .htaccess file:
Header set X-Robots-Tag "noodp"
This tells the server to include the following line in the response headers:
X-Robots-Tag: noodp
Search engines (that support X-Robots-Tag) will interpret this header line exactly the same way they would interpret 'noodp' in a robots meta tag. In general, you can put anything in an X-Robots-Tag header that you can put in a robots meta tag. Note that the page must not be blocked by robots.txt, otherwise the crawler will never request the page, and will therefore never see the header.