Search code examples
seorobots.txtmeta-tags

Meta tag vs robots.txt

  1. Is it better to use meta tags* or the robots.txt file for informing spiders/crawlers to include or exclude a page?

  2. Are there any issues in using both the meta tags and the robots.txt?

*Eg: <#META name="robots" content="index, follow">

Solution

  • Robots.txt IMHO.

    The Meta tag option tells bots not to index individual files, whereas Robots.txt can be used to restrict access to entire directories.

    Sure, use a Meta tag if you have the odd page in indexed folders that you want skipping, but generally, I'd recommend you most of your non-indexed content in one or more folders and use robots.txt to skip the lot.

    No, there isn't a problem in using both - if there is a clash, in general terms, a deny will overrule an allow.