How can I set a dynamic explicit wait using Selenium in Python?

Question

I got a little problem here with a program I built a few days ago, so I'm going to explain it very straightforward:

I'm scraping data from a page with my program, and for doing so, I set this Explicit Wait:

WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH,'/this_is_the_XPATH')))

For most of times, it works pretty well, however, there are sometimes in which the Explicit Wait above causes this error:

TimeoutException

Which makes my program stop and have to repeat a long process again.

I realized that the wait parameter set as 10 seconds is not always a good one, so I wanted to know from you guys if there's a way to set this parameter as a variable that always gets the exact load time (in seconds) for a page that is loaded completely.

Other more simple idea could be to repeat the process forcibly until the visibility of element is finally loaded, like by using try and exception blocks, but I don't know what should I type in the exception block to repeat the try one over and over until it's done.

As always, feedback is appreciated, thanks for reading in advance.

Answer 1

Always write reusable functions for repetitive actions and write exception-handling for those functions.

Your code:

 WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH,'elementTxt')))

Create a reusable function:

def clickElement(elementTxt, driver, time) 
    try :

      element = WebDriverWait(driver, time).until(EC.visibility_of_element_located((By.XPATH,'/this_is_the_XPATH')))
      element.click()

     except ElementClickInterceptedException as e: 
        driver.execute_script ("arguments[0].click();",element)

     except TimeoutException as te
     element = WebDriverWait(driver, time).until(EC.element_to_be_clickable((By.XPATH,'/this_is_the_XPATH')))
     element.click()
    

Call the function: Now, you can call this clickElement function wherever you need to perform click action. You need to pass the xPath value, driver instance and time in seconds.

clickElement(xPathText, driver, 30)

Other more simple idea could be to repeat the process forcibly until the visibility of element is finally loaded

If we know element will not be visible then we shouldn't wait till it displays. It cause the code to stay in infinite loop but you can search for element one more time. If you see above example, in TimeoutException block we are searching for same element again if still element is not accessible then we need think other ways.