Search code examples
searchweb-crawlersearch-engine

Hiding folders from search engines

How to mark folder hidden for all search engines?

How many alternatives are there?

Which one is the most reliable?

Solution

  • To prevent search engines from visiting certain directories/urls it is common practice to use robots.txt. This is a file that search engines take a look at before spidering your website.

    robots.txt

    User-agent: *
Disallow: /secret/

    This file should be placed in your website root. For example http://www.example.com/robots.txt

    There are two important considerations when using /robots.txt:

    • robots can ignore your /robots.txt. Especially malware robots that scan the web for security vulnerabilities, and email address harvesters used by spammers will pay no attention.
    • the /robots.txt file is a publicly available file. Anyone can see what sections of your server you don't want robots to use.