Search code examples
google-apps-scriptweb-crawlergooglebot

Google Apps Script Bot repeatedly crawling website each minute


Just an hour ago, I started tailing apache log file (access logs), and since then, I ve noticed this weird user agent, (couldnt find it Google's Official Docs).

I m feeling suspicious about it, since I couldnt find anything on Google's site about this User Agent,

("Mozilla/5.0 (compatible; GoogleApps script; +http://script.google.com/bot.html))

It has crawling the login page of our site, every minute, the whole day.

Here's the log snippet:

72.14.199.55 - - [07/Aug/2013:16:06:28 +0000] "GET / HTTP/1.1" 302 639 "-" "Mozilla/5.0 (compatible; GoogleApps script; +http://script.google.com/bot.html)"

72.14.199.55 - - [07/Aug/2013:16:06:28 +0000] "GET /accounts/login/ HTTP/1.1" 200 3780 "-" "Mozilla/5.0 (compatible; GoogleApps script; +http://script.google.com/bot.html)"

And it has been the same IP. And it still is,

Is it common to see this pattern of crawling?


Solution

  • Google Apps Scripts allow users to write up JavaScript based code and set them to run at specified intervals from Google's servers.

    This isn't Google's Web Crawler, it's a script that a Google User wrote.

    Also: If you block the IP or whatever the Google user will probably be bombarded with "Your script has failed to run" emails every minute.