Search code examples
web-applicationssearch-engineweb-crawler

Web crawlers and GET vs POST requests


I have heard that web crawlers are supposed to follow only GET requests and not POST ones.

In the real world is this a valid assumption?


Solution

  • Generally they do not do POST requests. This is just the current state of affairs and is not dictated anywhere, I believe. Some search engines are experimenting with crawling forms, but these are still GET requests.