I'm trying to make a webscraper for this website. The idea is that code iterates over all institutions by selecting the institution's name (3B-Wonen at first instance), closes the pop-up screen, clicks the download button, and does it all again for all items in the list.
However, after the first loop it throws the StaleElementReferenceException
when selecting the second institution in the loop. From what I read about it this implies that the elements defined in the first loop are no longer accessible. I've read multiple posts but I've no idea to overcome this particular case.
Can anybody point me in the right directon? Btw, I'm using Pythons selenium and I'm quite a beginner in programming so I'm still learning. If you could point me in a general direction that would help me a lot! The code I have is te following:
#importing and setting up parameters for geckodriver/firefox
...
# webpage
driver.get("https://opendata-dashboard.cijfersoverwonen.nl/dashboard/opendata-dashboard/beleidswaarde")
WebDriverWait(driver, 30)
# Get rid of cookie notification
# driver.find_element_by_class_name("cc-compliance").click()
# Store position of download button
element_to_select = driver.find_element_by_id("utilsmenu")
action = ActionChains(driver)
WebDriverWait(driver, 30)
# Drop down menu
driver.find_element_by_id("baseGeo").click()
# Add institutions to array
corporaties=[]
corporaties = driver.find_elements_by_xpath("//button[@role='option']")
# Iteration
for i in corporaties:
i.click() # select institution
driver.find_element_by_class_name("close-button").click() # close pop-up screen
action.move_to_element(element_to_select).perform() # select download button
driver.find_element_by_id("utilsmenu").click() # click download button
driver.find_element_by_id("utils-export-spreadsheet").click() # pick export to excel
driver.find_element_by_id("baseGeo").click() # select drop down menu for next iteration
See the problem here is, that you have defined a list corporaties = driver.find_elements_by_xpath("//button[@role='option']")
and then iterating
this list
, and clicking on first element
, which may cause some redirection to a new page, or in a new tab
etc.
so when Selenium
try to interact with the second webelement
from the same list
, it has to come back to original page
, and the moment it comes back, all the elements
become stale in nature
.
one of the basic solution in this cases are to define the list again, so that element won't be stale. Please see the illustration below :-
Code :
corporaties=[]
corporaties = driver.find_elements_by_xpath("//button[@role='option']")
# Iteration
j = 0
for i in range(len(corporaties)):
elements = driver.find_elements_by_xpath("//button[@role='option']")
elements[j].click()
j = j + 1 # select institution
driver.find_element_by_class_name("close-button").click() # close pop-up screen
action.move_to_element(element_to_select).perform() # select download button
driver.find_element_by_id("utilsmenu").click() # click download button
driver.find_element_by_id("utils-export-spreadsheet").click() # pick export to excel
driver.find_element_by_id("baseGeo").click() # select drop down menu for next iteration
time.sleep(2)