I have a website Flamingone.com. I registered my website on Google Webmaster some days ago. Today I checked what has Google Webmaster prepared about my website. Google said,
"Googlebot couldn't crawl your URL (/cgi-bin) (flamingone.com/cgi-bin), because your server either requires authentication to access the page, or it is blocking Googlebot from accessing your site.
But I neither have any folder named cgi-bin nor have a robots.txt. It should fire a 404 (My 404.php page). Why is it showing access denied?
Try to create a robots.txt on your root folder with the following content:
robots.txt
User-agent: *
Disallow: /cgi-bin/