I have a question regarding the way implicit wait works in selenium. As far as I understand implicit waits wait until an element is located/visible/present, with a given maximum. For instance:
wait = WebDriverWait(driver,
10).until(EC.presence_of_element_located((By.CLASSNAME,'classname')))
This statement makes selenium wait until the presence of the element with class name 'classname' is found, or until the maximum of ten seconds of waiting time is met, right?
Now, I've written a script that gets data from a website and uses implicit waits like this:
def move_to_next_page():
(this function loads the next page)
def get_page_data():
wait = WebDriverWait(driver,
10).until(EC.presence_of_element_located((By.CLASS_NAME, 'class')))
items = driver.find_elements_by_class_name('class')
for item in items:
itemList.append(item.text)
return itemList
move_to_next_page()
get_page_data()
Yesterday, I ran this script successfully several times; the implicit waits paused my program for up to five seconds to ensure everything went right. However, I'm trying to run the script right now and about 70% of the time I get an error message saying:
selenium.common.exceptions.StaleElementReferenceException: Message:
stale element reference: element is not attached to the page document
Implying that browser is still loading? The weird thing is that I get this message way before the limit of 10 seconds is met. I've even tried 20 and 30 seconds but selenium still crashes a lot of times. Why won't selenium wait for at least 10/20/30 seconds?
I'm pretty sure implicit waits cause the crashes, since when I'm using explicit waits like:
time.sleep(4)
The program runs every time.
I have the data I'm looking for so I don't really need this script any longer. It's just that it's really frustrating not to be able to write something that will work regardless of the browser loading time.
First of all, WebDriverWait and ExpectedConditions are explicit wait, not implicit wait. You can learn more about the difference here, but long story short explicit wait is waiting for certain condition to be met and implicit wait is waiting for the element to exist in the DOM.
As for the exception, StaleElementReferenceException doesn't mean the page is not loaded, it means the DOM has changed or reloaded between the time you located the element and the time you tried to do something with it. You can see it in the error message
stale element reference: element is not attached to the page document
You can try presence_of_all_elements_located instead
items = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CLASS_NAME, 'class')))
for item in items:
itemList.append(item.text)
WebDriverWait.until will return the elements the ExpectedConditions was checking.