Using python request module(get function) I am scraping the links i.e. a crawler. I make multiple requests using the script. Since I was giving too many requests, Google intervened with CAPTCHA which was reset after sometime. I am using time module to put code to sleep for some time after each request to avoid it. I want to know what will be the apt time to put the script to sleep so that Google doesnt see it as invalid traffic ? (I am using a college proxy server)
I founded this sites very helpful for correct crawling: h
Enjoy!