Search code examples
pythonselenium-webdriverwebdriverwait

What is the best way to handle waiting for elements that are not always present with Selenium?


For example if I have a code like this:

<div id='parent'>
  <div class='profile-pic'>...</div>
  <p class='email'>I'm not always present</p>
  <span class='name'>John Smith</span>
</div>

Now this is a random example. But should help to visualize what I mean. I want to grab profile picture, email and the name. None of them are necessarily always present on the page.

I could set a wait for each element separately like this:

try:
  wait = WebDriverWait(driver, 10)
  wait.until(EC.visibility_of_element_located((By.CLASS_NAME, 'email')))
except:
  pass

But that would mean every time one or more elements are not present on the page I would wait 10 seconds for each of them. Now let's say I have 20 elements like this that can be present but are not always and I have 1000 pages to go through. This approach would take forever.

Now that's where my question comes in. What is the best solution to handle cases like this? Or is this something that should not be done with Selenium at all?

What I have been doing so far is selecting the parent element and waiting for it to load but as I have learned that does not guarantee that all the children are loaded.


Solution

  • What you can do here is to iterate over parent elements.
    For each parent element wait for it's presence.
    I can't know if parent element here is visible or not.
    In case parent element is visible - wait for it visibility.
    Now, wait for desired child element visibility. Here you can use a short timeout since this timeout is for use in case the parent element is already present (but maybe still not fully loaded) while child element is still need to be loaded.
    2-3 seconds should be more that enough here until you have very bad internet connectivity / loading is done extremmely low.