Displaying sites every 10 seconds with Python and Selenium from a CSV

Question

I am doing an automation with Python. This automation should take a list of websites from a .csv file and perform click activities on them, which are working correctly, so I will omit parts of the code.

I adjusted a line of code which is this one:

if time.time() - start_time > 10:
    print("Time limit exceeded, moving to the next site")
    break

This line is responsible for moving to the next site in the list after 10 seconds, regardless of whether all activities have been completed or not. The robot should behave this way, performing the activities within a time limit, without needing to successfully complete all activities.

All sites are working correctly, except for the w3schools site, which freezes and does not move to the next one. All sites should remain open for a maximum of 10 seconds.

How can I force certain sites to move to the next one? This automation needs to be dynamic, as it will receive a .csv spreadsheet with 600 sites.

Below is the code for the automation:

import time
import csv
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import NoSuchElementException, ElementNotInteractableException
from webdriver_manager.chrome import ChromeDriverManager

# Install a specific version of ChromeDriver
driver_path = ChromeDriverManager(driver_version="125.0.6422.77").install()

if __name__ == "__main__":
  
    # Selenium WebDriver configuration
    chrome_options = Options()
    chrome_options.add_argument("--start-fullscreen")  # Open the browser in fullscreen mode

    driver = webdriver.Chrome(service=Service(driver_path), options=chrome_options)

    # Reading the CSV file
    with open('websites2.csv', newline='') as csvfile:
        reader = csv.DictReader(csvfile)
        for row in reader:
            website = row['website']
            try:          
                driver.get(website)

                start_time = time.time()  # Capture the start time               

                if time.time() - start_time > 10:
                    print("Time limit exceeded, moving to the next site")
                    break

            except Exception as e:
                print(f"Error processing {website}: {e}")

    # Close the browser at the end of the process
    driver.quit()

Answer 1

My previous answer was erroneous; a simpler solution is to set a timeout on the driver, so it will never get stuck loading a page forever.

After creating the driver, add a set_page_load_timeout on the driver:

driver = webdriver.Chrome(service=Service(driver_path), options=chrome_options)
driver.set_page_load_timeout(timeout)

With that you can completely remove the if time.time() - start_time > 10 block.