Does anyone know how to detect whether an incoming request is from a search engine robot? Do HTML headers contain any specific info for that?
You can usually use a combination of IP and the user agent in the HTTP headers to figure it out. There is an article all about it here: http://www.jafsoft.com/searchengines/spider_hunting.html