I have been working on a web scraper in python to scrape Google Finance, but I can't find the specific tag I'm looking for using the find() method. Finally, I got so annoyed that I decided to write the returned data to a file and look for it myself. So I wrote it to testing.html in the same directory, and opened it with Google Chromium so I could use the inspect tool. Within minutes, I found the element I was looking for. What am i doing wrong? My code is attached below:
import dryscrape
session = dryscrape.Session()
def get(url):
global session
try:
session.visit(url)
data = session.body()
except:
print('Connection Failed')
return str(data)
def save(price, stockname):
pass
def extract(data):
return data.find('<div class="YMLKec fxKbKc">')
class following():
apple = "https://www.google.com/finance/quote/AAPL:NASDAQ"
tesla = "https://www.google.com/finance/quote/TSLA:NASDAQ"
google = "https://www.google.com/finance/quote/GOOGL:NASDAQ"
amazon = "https://www.google.com/finance/quote/AMZN:NASDAQ"
microsoft = "https://www.google.com/finance/quote/MSFT:NASDAQ"
netflix = "https://www.google.com/finance/quote/NFLX:NASDAQ"
def __init__():
global apple
global tesla
global google
global amazon
global microsoft
global netflix
save(extract(get(following.apple)), following.apple)
save(extract(get(following.tesla)), following.tesla)
save(extract(get(following.google)), following.google)
save(extract(get(following.amazon)), following.amazon)
save(extract(get(following.microsoft)), following.microsoft)
save(extract(get(following.netflix)), following.netflix)
f = open("testing.html")
print(extract(f.read()))
f.close()
Found the issue: It's not YMLKec but YMlKec. Not a capital L.
data = open("testing.html", "r").read()
class_ = "YMlKec fxKbKc"
print(data.find(class_))
>>> 992880