Search code examples
robots.txtgoogle-custom-search

Allow only Google CSE and disallow Google standard search in ROBOTS.txt

I have a site that I am using a Google Custom Search Engine on. I want Google CSE to crawl my site but I want it to stay out of the results of a regular Google search. I put this in my robots.txt file hoping that google CSE bots would ignore it since I specified the pages I wanted Google CSE to crawl in the settings 

User-agent: *
Disallow: /

I guess the Google CSE bots also have to obey robots.txt. So is there a way to get my page to stay out of search engine searches but for Google CSE to still be able to index it? TIA!

Solution

  • There is no solution to this question that fits with what you would like. I too have the same situation where I need custom search only. Unfortunately Google's list of crawlers doesn't show a specific bot for google custom search. Blocking Googlebot will both kill native search and custom search.