Search code examples
pythonhtmlurlgethttp-status-code-404

Python requests.get(URL) returns 404 error when using URL with dot

I'm trying to get webpage https://finance.yahoo.com/quote/AFLT.ME using Requests library for Python.

This link opens well in browser, but results in Error 404 while using this code:

import requests
r = requests.get('https://finance.yahoo.com/quote/AFLT.ME')

I'm pretty sure that the problem is in the dot (.) symbol in the "AFLT.ME" as code works well with URLs without dot - for example https://finance.yahoo.com/quote/AAPL

I already have found answers solving this problem BUT on website owner side.

But how can I solve this issue doing GET requests?

I have tried some advises that unfortunately DID NOT help:

  • to replace dot . with %2f like /AFLT%2EME
  • to add slash / in the end like /AFLT.ME/
Solution

  • Strange, it seems that if one sends the User-Agent header, even with an empty value, it then responds with a 200:

    >>> requests.get('https://finance.yahoo.com/quote/AFLT.ME', headers={'User-Agent': ''})
<Response [200]>

    Edit: The same issue was reported here: https://stackoverflow.com/a/68259438/9835872