Search code examples
web-crawlergoogle-crawlers

robots.txt to remove searche engine results


I heard you can remove google searches & images by adding something to robots.txt in main folder that the web directs to, how do I do that and how long does it take for the content to disapear?


Solution

  • User-agent: *
    Disallow: /
    

    First line tells : Apply following to all the robots. Second line tells : Dont allow any page to be parsed.