Search code examples
pythonseleniumdebuggingselenium-webdriverfirebug

Selenium Webdriver Timeout (Python 2.7)


When scraping data from NASDAQ there are tickers like ACHC that have empty pages. ACHC Empty Field

My program iterates through all ticker symbols and when I get to this one it times out because there is no data to grasp. I am trying to figure out a way to check if there is nothing and if so skip the ticker, but continue the loop. The code is pretty long, so Ill post the most relevant part: the beginning of the loop where it opens the page:

## navigate to income statement annualy page    
url = url_form.format(symbol, "income-statement")
browser.get(url)

company_xpath = "//h1[contains(text(), 'Company Financials')]"
company = WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.XPATH, company_xpath))).text

annuals_xpath = "//thead/tr[th[1][text() = 'Period Ending:']]/th[position()>=3]"
annuals = get_elements(browser,annuals_xpath)

Here is a pic of the error message


Solution

  • Selenium doesn't have a built-in method for determining whether an element exists or not, so the most common thing to do is use a try/except block.

    from selenium.common.exceptions import TimeoutException
    ...
    try:
        company = WebDriverWait(browser, 10).until(EC.presence_of_element_located((By.XPATH, company_xpath))).text
    except TimeoutException:
       continue
    

    This should keep the loop going without crashing, assuming that continue works as expected with your loop.