Search code examples
curluser-agent

Why does www.amazon.com render binary data?

If you curl www.amazon.com the server responds with binary data.

curl -L https://www.amazon.com | less

Yet, when you add a user agent or look at the exact same url in the browser, they send valid HTML markup.

curl -L -A "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/534.30 (KHTML, like Gecko) Chrome/12.0.742.112 Safari/534.30" https://www.amazon.com | less

Why would amazon choose to send binary data to clients that don't have a user agent specified? I'm not sure what they have to be gained by doing that.

Sure, they could be trying to obfuscate their responses to bots. But if that's the case, they might as well not send a response at all.

Any insight would be appreciated.

Solution

  • Check the content encoding header with the ‘-i’ flag. The value is most likely gzip, which means thr content is a compressed binary that most browsers will decode upon receipt.