Search code examples
nginxservergooglebot

getting hits on my web server from google bots

I am not able to understand why i am seeing these logs into my nginx server.

66.249.79.115 - - [06/Oct/2015:18:50:17 +0000] "GET /profile/?Rohatgi.Nikhil HTTP/1.1" 404 1031 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.115 - - [06/Oct/2015:18:50:49 +0000] "GET /profile/?Mukherjee.PankajKumar HTTP/1.1" 404 1038 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.115 - - [06/Oct/2015:18:51:21 +0000] "GET /profile/?Khorana.Ashish HTTP/1.1" 404 1031 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.117 - - [06/Oct/2015:18:51:52 +0000] "GET /profile/?Mittal.AshokKumar HTTP/1.1" 404 1034 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.117 - - [06/Oct/2015:18:52:24 +0000] "GET /profile/?Suri.Divya HTTP/1.1" 404 1029 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.79.117 - - [06/Oct/2015:18:52:56 +0000] "GET /profile/?gupta.member) HTTP/1.1" 404 1030 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

I made my website live very recently.

From logs i can understand little bit that they are coming from Google bot but i want to understand why i am getting this and how can i block them ? If i will block there will be no harm to my website.

Solution

  • This behavior is normal if you put your site live recently. Google Bots are crawling the web and are indexing new webpages, so they can be found using google search. As you can see from the logifle, the google bot pretends to be an iPhone (iPhone; CPU iPhone OS 8_3 like Mac OS X).

    If you want to block the googlebot, you can follow this guide: Nginx + CDN + GoogleBot or how to avoid many useless Googlebot hits Please note that your webpage will not be found using google search if you block the google crawler bot. If you want to block wider range of spider/crawling bots, please refer to the post by user GD-hussle.

    For a general information on google crawlers take a look at Crawling, indexing & ranking.