Search code examples
search-engineweb-crawlerrobots.txt

Google indexed my test folders on my website :( How do I restrict the web crawlers!


Help Help! Google indexed a test folder on my website which no one save I was supposed to know about :(! How do I restrict google from indexing links and certain folders.


Solution

  • Use a robot exclusion file, or better yet password protect your test areas! Using a robots.txt file to "protect" areas you don't want others to see is a little like hanging a sign on your back door saying "I've left this open but please don't come in" :)

    If you sign up for Google webmaster tools, you can request removal of a search result if you ensure it's no longer accessible by their crawler.