Getting selenium to work on pythonanywhere

Question

My understanding is that pythonanywhere supports a headless Firefox browser but you need

from pyvirtualdisplay import Display

And so you can connect using

with Display():
    while True:
        try:
            driver = webdriver.Firefox()
            break
        except:
            time.sleep(3)

And I connect just fine. However, after I start using the driver with

with Display():
    while True:
        try:
            driver = webdriver.Firefox()
            break
        except:
            time.sleep(3)
    wb=load_workbook(r'/home/hoozits728/mutual_fund_tracker/Mutual_Fund_Tracker.xlsx')
    ws=wb.get_sheet_by_name('Tactical')

    for i in range(3, ws.max_row+1):
        if ws.cell(row=i,column=2).value is not None:
            driver.get('https://finance.yahoo.com/quote/' + ws.cell(row=i,column=2).value + '/performance?ltr=1')
            oneyear=driver.find_element_by_css_selector('#Col1-0-Performance-Proxy > section > div:nth-child(2) > div > div:nth-child(5) > span:nth-child(2)').text
            threeyear=driver.find_element_by_css_selector('#Col1-0-Performance-Proxy > section > div:nth-of-type(2) > div > div:nth-of-type(6) > span:nth-of-type(2)').text
            fiveyear=driver.find_element_by_css_selector('#Col1-0-Performance-Proxy > section > div:nth-of-type(2) > div > div:nth-of-type(7) > span:nth-of-type(2)').text
            ws.cell(row=i,column=10).value=oneyear
            ws.cell(row=i,column=11).value=threeyear
            ws.cell(row=i,column=12).value=fiveyear

           … and so on …

I get this error after just a little while

For what it's worth, this code works perfectly fine on my local machine. Also, I am a paying member, so there should be no whitelist issues.

Answer 1

It has recently come to my understanding that yahoo has blocked pythonanywhere from running any web scraping scripts. I assume this is true for all AWS servers and those who use them, but I am not 100% certain of this. I hope this helps anyone who comes across this question.

https://www.pythonanywhere.com/forums/topic/5724/#id_post_52307