Search code examples
python-3.xweb-scrapingstockstockquotes

web-scraping vwap value for list of stock in data frame

scraping particular value (vwap) from nse web site: here i have a data frame with stock list , i need to fetch the vwap value for every stock from nse website . below is the reproducibe code .

    stock_list = ['SKIPPER','NIPPOBATRY','RANEHOLDIN','OSWALAGRO','GINNIFILA','VOLTAMP','NACLIND','GALLANTT','ASAHISONG','KSL','UNICHEMLAB',
'TRENT','TIL','MMP','SHARDAMOTR','ARCHIES','MAGADSUGAR']
df = pd.DataFrame()
df['stock_list'] = stock_list
df['vwap_value'] = 0

As shown in the image highlited with yellow area to fetch and insert into respective vwap column

How to scrape vwap value again the stock name specified in the data frame

https://www1.nseindia.com/index_nse.htm

so In the above url once this url is called after that SKIPPER from the first stock_list would be entered in the EQUITY search box and new url page be navigated

https://www1.nseindia.com/live_market/dynaContent/live_watch/get_quote/GetQuote.jsp?symbol=SKIPPER&illiquid=0&smeFlag=0&itpFlag=0 and then vwap value would be fetch from the new link.

Solution

  • The data is embedded within the page in Json format, so BeautifulSoup doesn't see it. You can use json module to load it.

    For example:

    import json
import requests
from bs4 import BeautifulSoup


url = 'https://www1.nseindia.com/live_market/dynaContent/live_watch/get_quote/GetQuote.jsp?symbol=SKIPPER&illiquid=0&smeFlag=0&itpFlag=0'
headers = {'User-Agent': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:77.0) Gecko/20100101 Firefox/77.0'}

soup = BeautifulSoup(requests.get(url, headers=headers).content, 'html.parser')
data = json.loads(soup.select_one('#responseDiv').text)

# uncomment this to print all data:
# print(json.dumps(data, indent=4))

print('averagePrice:', data['data'][0]['averagePrice'])

    Prints:

    averagePrice: 45.52