Search code examples
web-crawlerhttprequestgooglebot

Is it possible to use Googlebot's user agent token to detect it rather than the full user agent string?

From: https://support.google.com/webmasters/answer/1061943?hl=en

The following table shows the crawlers used by various products and services at Google:

  • User agent token is used in the User-agent: line in robots.txt to match a crawler type when writing crawl rules for your site. Some crawlers have more than one token, as shown in the table; you need to match only one crawler token for a rule to apply. This list is not complete, but covers most of the crawlers you might see on your website.

  • Full user agent string is a full description of the crawler, and appears in the request and your web logs.

enter image description here

QUESTION

From the excerpt above we can see that it's possible to use the User agent token inside the robots.txt file to match and therefore detect a crawler.

I would like to use the User agent token on my server to detect Googlebot crawler requests. So I don't have to hard code the full user agent string.

But is the User agent token present in the request headers? Is it possible to use it or should I stick to the full user agent string?

Solution

  • If you compare the values on the columns "User agent token" and "Full user agent string" you can see the "product token" is also available in the "Full user agent string". So you can check if the "Full user agent string" contains the "User agent token".

    The numbers on the "Full user agent string" will change in future more often than the "produkt token".