Search code examples
pythonseleniumstaleelementreferenceexception

StaleElementReferenceException while looping over list


I'm trying to make a webscraper for this website. The idea is that code iterates over all institutions by selecting the institution's name (3B-Wonen at first instance), closes the pop-up screen, clicks the download button, and does it all again for all items in the list.

However, after the first loop it throws the StaleElementReferenceException when selecting the second institution in the loop. From what I read about it this implies that the elements defined in the first loop are no longer accessible. I've read multiple posts but I've no idea to overcome this particular case.

Can anybody point me in the right directon? Btw, I'm using Pythons selenium and I'm quite a beginner in programming so I'm still learning. If you could point me in a general direction that would help me a lot! The code I have is te following:

#importing and setting up parameters for geckodriver/firefox
...

# webpage
driver.get("https://opendata-dashboard.cijfersoverwonen.nl/dashboard/opendata-dashboard/beleidswaarde")

WebDriverWait(driver, 30)

# Get rid of cookie notification
# driver.find_element_by_class_name("cc-compliance").click()

# Store position of download button
element_to_select = driver.find_element_by_id("utilsmenu")
action = ActionChains(driver)
WebDriverWait(driver, 30)

# Drop down menu
driver.find_element_by_id("baseGeo").click()

# Add institutions to array
corporaties=[]
corporaties = driver.find_elements_by_xpath("//button[@role='option']")

# Iteration
for i in corporaties:
    i.click()                                                     # select institution
    driver.find_element_by_class_name("close-button").click()     # close pop-up screen
    action.move_to_element(element_to_select).perform()           # select download button
    driver.find_element_by_id("utilsmenu").click()                # click download button
    driver.find_element_by_id("utils-export-spreadsheet").click() # pick export to excel
    driver.find_element_by_id("baseGeo").click()                  # select drop down menu for next iteration

Solution

  • Problem Explanation :

    See the problem here is, that you have defined a list corporaties = driver.find_elements_by_xpath("//button[@role='option']") and then iterating this list, and clicking on first element, which may cause some redirection to a new page, or in a new tab etc.

    so when Selenium try to interact with the second webelement from the same list, it has to come back to original page, and the moment it comes back, all the elements become stale in nature.

    Solution :

    one of the basic solution in this cases are to define the list again, so that element won't be stale. Please see the illustration below :-

    Code :

    corporaties=[]
    corporaties = driver.find_elements_by_xpath("//button[@role='option']")
    # Iteration
    j = 0
    for i in range(len(corporaties)):
        elements = driver.find_elements_by_xpath("//button[@role='option']")
        elements[j].click()
        j = j + 1 # select institution
        driver.find_element_by_class_name("close-button").click()     # close pop-up screen
        action.move_to_element(element_to_select).perform()           # select download button
        driver.find_element_by_id("utilsmenu").click()                # click download button
        driver.find_element_by_id("utils-export-spreadsheet").click() # pick export to excel
        driver.find_element_by_id("baseGeo").click()  # select drop down menu for next iteration
        time.sleep(2)