Search code examples
search-enginemeta-tagsrobots.txt

Will this robot.txt code disallow every Search Engine?

I would like to know if this code disallow every search engine to scan my directory.

User-agent: * 
Disallow: /

also does this code is updated with the new htlm 5 protocole ?

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

Really useful or not needed anymore?

Solution

  • No, it'll only stop those that obey the robots.txt protocol (there exist search engines that intentionally disobey robots.txt to find "hidden" stuff).

    It will however stop the vast majority of search engines that your average consumer would use, from seeing it.