Search code examples
python-3.xseleniumselenium-webdriverchrome-web-driver

Can't start more than 17 Selenium-webdrivers

I'm using Selenium with ChromeDriver in a python multithreading environment. Before multiprocessing starts, each browser-instance is setup. It is possible to start 17 browsers without problem, but if it is more than 17 instances it fails.

Here are the log outputs of the browsers webdriver.Chrome(service_log_path=self.session_logfile):

working instance (instance #17):

[…]
[1640642658.135][INFO]: [d95aa29866064789c167e42c4911ed36] RESPONSE SetTimeouts
[1640642658.136][INFO]: [d95aa29866064789c167e42c4911ed36] COMMAND Navigate {
   "url": "https://www.website.com"
}
[1640642658.136][INFO]: Waiting for pending navigations...
[1640642658.137][INFO]: Done waiting for pending navigations. Status: ok
[1640642658.417][INFO]: Waiting for pending navigations...
[1640642659.122][INFO]: Done waiting for pending navigations. Status: ok
[1640642659.122][INFO]: [d95aa29866064789c167e42c4911ed36] RESPONSE Navigate
[1640642659.127][INFO]: [d95aa29866064789c167e42c4911ed36] COMMAND DeleteAllCookies {

}
[1640642659.127][INFO]: Waiting for pending navigations...
[1640642659.128][INFO]: Done waiting for pending navigations. Status: ok
[…]

failing instance (instance #18):

[…]
[1640642194.176][INFO]: [9148f58db0e76ea70186f0739b04e677] RESPONSE SetTimeouts
[1640642194.176][INFO]: [9148f58db0e76ea70186f0739b04e677] COMMAND Navigate {
   "url": "https://www.website.com"
}
[1640642194.176][INFO]: Waiting for pending navigations...
[1640642194.178][INFO]: Done waiting for pending navigations. Status: ok
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00
[1640642194.122][INFO]: [330dd18a9973da505814eddae0cfd97d] COMMAND QuitAll {

}
[1640642194.172][INFO]: [330dd18a9973da505814eddae0cfd97d] RESPONSE QuitAll
  • The issue is not about multi-threading, since I sequentially start the instances, before I work with parallel processess.
  • The machine has more resources, so it is not about a limitation of CPU or memory.
  • On another machine I can run 30 instances without problems.

I tried debugging it, but can't understand why it happens. How can I retrieve more information on why this instance fails/quits?

Solution

  • If you're going to be using multithreading for Python tests, you may consider using "pytest-xdist" to handle the multiple processes for you: https://github.com/pytest-dev/pytest-xdist. And if you're using Selenium with Python and pytest, there's a framework that may simplify the Selenium test multithreading for you, "SeleniumBase": https://github.com/seleniumbase/SeleniumBase. It functions as a pytest plugin, so you can use the pytest multi-threading args provided by pytest-xdist, and run all your Selenium Python tests multithreaded as needed. Eg: pytest -n 20 for 20 parallel threads, assuming your machine has enough memory for that. If your machine doesn't have enough memory for that, then definitely use fewer threads. Spinning up a lot of browser windows can be very memory-intensive.