Search code examples
selenium-webdrivercelerypython-multithreading

"Connection pool is full, discarding connection" warning Selenium on a Celery Task in Django


So I'm using Celery on Django to schedule tasks. One of my tasks generate a driver with Selenium that drive a chrome node on Selenium Grid.

The task also spawns a Thread that takes the driver as args with the only goal to close the pop-up on the page. It basically runs infinitely looking for the pop-up to close

options = webdriver.ChromeOptions()
options.add_argument('--ignore-ssl-errors=yes')
options.add_argument('--ignore-certificate-errors')
options.add_experimental_option("prefs", 
                                    {"profile.content_settings.exceptions.clipboard": {"[*.]mywebsite.com,*": 
                                                                                       {'last_modified': (time.time()*1000), 'setting': 1}}})

    driver = webdriver.Remote(
            command_executor='http://localhost:4444/wd/hub',
            options=options
        )

stop_event= threading.Event()

    popup_thread = threading.Thread(
                target=close_popup_window,
                args=(driver,
                      stop_event)
            )

    popup_thread.daemon = True
    popup_thread.start() 

The function of the Thread is

def close_popup_window(driver, stop_event):
    list_popup = ["/html/body/div[3]/div[3]/div/div[1]/div[1]/div[2]/button",
                  "//*[@id=\"max-width-dialog-title\"]/h2/button",
                  "/html/body/div[6]/div[3]/div/div[1]/button",
                  "/html/body/div[5]/div[3]/div/div[1]/button",
                  "/html/body/div[3]/div[3]/div/div[1]/button",
                  "/html/body/div[4]/div[3]/div/div[1]/button",
                  "//*[@id=\"onetrust-accept-btn-handler\"]",
                  "//*[@id=\"max-width-dialog-title\"]/button",
                  ]
    while not stop_event.is_set():
        for popup in list_popup:
            try:
                close_button = driver.find_element(By.XPATH, popup)
                close_button.click()
            except:
                pass

The main program of the task, the one that starts the thread, uses the driver as well like

text_box = driver.find_element(By.XPATH,"//*[@id=\"inputText\"]")
text_box.send_keys(first_half)

It seems the thread generates the Warning connection pool is full, discarding connection. Everything seems working fine, I just have this warning popping up on the Celery console but I can't understand why it is generated and how fix it.

I have tried


Solution

  • As suggested WebDriverWait helped fix the issue

    from selenium.webdriver.support.ui import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC
    from selenium.webdriver.common.by import By
    from selenium.common.exceptions import TimeoutException
    

    and I've used for instance

    text_box = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH,"the_actual_X_path")))
    

    or

    copy_button = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH,
                                        "the_actual_X_path")))