Search code examples
pythonweb-scrapingbeautifulsouphtml-parsingfindall

inconsistent behavior of beautifulSoup


I am trying to make a python file which gives real time stock updates. I tried for l in code.findAll('span', {'id' : 'ChangePercent'}): for :

<span id="ChangePercent">-0.71%</span>

it worked, source : money.rediff.com/ but using same technique on yahoo finance doesn't work.. i.e.

price = code.findAll('span', {'class' : 'yfi-price-change-green'})

fails to find : <span class="yfi-price-change-green">(1.95%)</span>

source : finance.yahoo.com/

the two codes : rediff : http://ideone.com/kslILJ yahoo : http://ideone.com/egGQLv


Solution

  • The values you are looking for on the Yahoo Finance Page are updated periodically via a streaming API call in the browser. You would not get these values by just requesting the Reliance Industries Ltd (RELIANCE.NS) url using urllib or requests.

    The least complex option would be to automate a real browser using selenium:

    >>> from selenium import webdriver
    >>>
    >>> driver = webdriver.Chrome()
    >>> driver.get('https://in.finance.yahoo.com/q?s=RELIANCE.NS&ql=0')
    >>> for element in driver.find_elements_by_class_name('yfi-price-change-green'):
    ...     print(element.text)
    ... 
    0.55%
    0.40%