Search code examples
httpwebrequestscreen-scraping

Is there any detectable difference between Browser and programmatic http requests

Is there any way to tell from headers or other data, whether a request is from a browser or non-browser program?

Solution

  • The browser is a programmatic HTTP request.
    It just happens to have a user in front of it.

    Some programmatic HTTP requests won't include a User-Agent header, though.

    You could do something fancy, such as sending recorded mouse movements using Javascript and checking them statistically, and the non-browser client can simulate them.

    Or you could just use a CAPTCHA.