Search code examples
pythonweb-scrapingbeautifulsoupyahoo-finance

Scraping from yahoo finance with beautifulsoup resulting in status code : 404

I am trying to scrape some data from yahoo finance using beautifulsoup, but I've run into a problem. I am trying to run the following code,

import xlwings as xw
import requests
import bs4 as bs

r = requests.get('https://finance.yahoo.com/quote/DKK=X?p=DKK=X&.tsrc=fin-srch')
soup = bs.BeautifulSoup(r.content,'lxml',from_encoding='utf-8')

However, when inspecting my output from "soup", I get the following status code in the section,

<body>
<!-- status code : 404 -->
<!-- Not Found on Server -->

I've run the exact same piece of code on another trading pair on yahoo finance with no problem whatsoever. Could anyone tell me what I am doing wrong?

Thanks in advance!

Solution

  • You need to inject user agent to get 200 response.

    #import xlwings as xw
import requests
import bs4 as bs

headers={'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.102 Safari/537.36'}
r = requests.get('https://finance.yahoo.com/quote/DKK=X?p=DKK=X&.tsrc=fin-srch',headers=headers)
print(r)
soup = bs.BeautifulSoup(r.content,'lxml')

    Output: <Response [200]>