Does Google's bot send HEAD requests as well as GET requests?
I wondering about this because I'm running a Tornado application that does not support HEAD request and I have a lot of 404 errors in Google Search Console.
To verify that the Tornado application is not accepting HEAD requests I do:
curl -I example.com/this-url-exists
HTTP/1.1 404 Not Found
Date: Thu, 10 Mar 2016 08:49:07 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 7520
Connection: close
X-Ua-Compatible: IE=edge,chrome=1
Server: TornadoServer/4.2.1
I suspect that the Googlebots are doing the same for some of the crawling.
Can this be right?
As far as I know, Googlebot does not send HEAD requests (and neither do other major search engines' crawlers). But either way, this 404 is not caused by the fact that the request is a HEAD: sending HEAD to a resource that does not support it returns status code 405, not 404.