TLDR: I create multiple selenium drivers for testing in different functions and that uses too much memory. Is there a way to limit memory usage of selenium? (I am already running headless)
So I am using selenium with pytest to run tests on my website. For seting up the corect environment for these tests to run I use multiple functions/fixtures that create another driver inside them and close the driver when they are done.
But this approach is problematic for me as it uses too much memory and crashes my machine each time I try to run/debug. I have several ultility functions in different files that are used by my tests and create their own driver to do their jobs. What I want to know is if I can use a single driver to do these ultility functions.
Here is the tests I try to run.
from lib.driver_options import get_driver_options
import lib.resetEnvr as resetEnvr
import lib.sayTRUST_ClientGroup as groupmanager
#... other imports
@pytest.fixture(scope="module")
def load_config():
file_path = os.path.dirname(__file__)
config_file_path = os.path.join(os.path.dirname(os.path.dirname(file_path)), 'config', 'website_config.json')
vz_config_path = os.path.join(os.path.dirname(os.path.dirname(file_path)), 'config',
'cloud_testserver.json')
with open(config_file_path) as f:
configurations = json.load(f)
with open(vz_config_path) as f:
vz_config = json.load(f)
default_config = configurations['default_config']
keyfile = os.path.join(os.path.dirname(os.path.dirname(file_path)), 'config', 'Service_PrivateKey_RSA')
return {
'default_config': default_config,
'vz_config': vz_config,
'keyfile': keyfile
}
class Test_NetworkAccess():
"""
Test cases:
Changing the network settings and making sure they all work.
"""
NATBUDDY_PRIVIP = "1.2.3.4"
NATBUDDY_PUBIP = "1.2.3.4"
TCPPORT = 8360
UDPPORT = 8361
config =None
@pytest.fixture(scope="function")
def setup_environment(self, load_config):
print("setup method")
options = get_driver_options()
service = Service()
driver = webdriver.Chrome(service=service, options=options)
driver.implicitly_wait(2)
vars = {}
# env reseter cleans up the websites state after each testing.
env_reseter = resetEnvr.dbCleaner()
ifaces = load_config['vz_config']["TemplateVM_info"]["network interfaces"]
for iface in ifaces:
for fixed_ip in iface.get("fixed_ips", []):
ip = fixed_ip.get("ip_address", "")
if ip.startswith("10.236."):
pub_ip = ip
else:
priv_ip = ip
# try to delete previous settings.
try:
groupmanager.delete_testClients()
except:
pass
try:
groupmanager.delete_testGroup()
except:
pass
yield {
'driver': driver,
'env_reseter': env_reseter,
'pub_ip': pub_ip,
'priv_ip': priv_ip,
'vars': vars
}
# Teardown after each test function
driver.close()
driver.quit()
@pytest.fixture(scope="module", autouse=True)
def setup_iface(self):
print("setup interface")
print("add_listenPort")
groupmanager.add_listenPort()
print("add_private_iface")
groupmanager.add_private_iface()
env_reseter = resetEnvr.dbCleaner()
print("get backup")
env_reseter.getBackup()
@pytest.fixture(scope="function")
def create_test_group(self,request, setup_environment):
print("create test group")
params = request.param
groupmanager.create_testGroup(**params)
print("create test group yields")
yield
# Teardown
groupmanager.delete_testGroup()
@pytest.fixture(scope="function")
def create_test_client(self, request, create_test_group):
params = request.param
print("create test client")
groupmanager.create_testClient(**params)
print("create test client yields")
yield
# teardown
groupmanager.delete_testClients()
@pytest.mark.parametrize("create_test_group", [
{"NATBuddyPub": False, "TCP": True, "UDP": False, "SSH": True},
{"NATBuddyPub": True, "TCP": True, "UDP": False, "SSH": True},
# Add other configurations as needed
], indirect=True)
@pytest.mark.parametrize("create_test_client", [
{},
], indirect=True)
@pytest.mark.usefixtures("setup_environment","create_test_group","create_test_client")
def test_NATBuddyGroupPubSSH(self,setup_environment,create_test_group,create_test_client, record_xml_attribute ,request):
# do some testing without using Driber
print("do some testing")
Here is an example how other ultility functions that do use the selenium driver And the setttings I use for the driver. They all use the driver in the same way.
def get_driver_options():
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
options.add_argument("--window-size=1440, 900")
options.add_argument('--ignore-certificate-errors')
options.add_argument('--allow-running-insecure-content')
options.add_argument("--disable-extensions")
options.add_argument("--proxy-server='direct://'")
options.add_argument("--proxy-bypass-list=*")
options.add_argument("--start-maximized")
options.add_argument('--disable-gpu')
options.add_argument('--disable-dev-shm-usage')
options.add_argument("--FontRenderHinting[none]")
options.add_argument('--no-sandbox')
options.add_argument('log-level=3')
options.add_argument('--ignore-ssl-errors=yes')
options.add_argument('--allow-insecure-localhost')
return options
def add_private_iface():
options = get_driver_options()
service = Service()
driver = webdriver.Chrome(service=service, options=options)
driver.implicitly_wait(15)
website_url = default_config["website_url"]
driver.get(website_url)
# do some stuff
# add some private interface to website
driver.find_element(By.LINK_TEXT, "Logout").click()
driver.close()
driver.quit()
And when I run these tests I observe that Google chrome stars using crazy amount of CPU and my machine freezes.
I have tried previously to run these functionss with passing the driver as a parameter but it doesn't seem to work out and gives bunch of strange errors. After trying to make it work I decided to do it this way but now I have doubts. Is it possible to make a single selenium driver for all of these?
Or maybe there is something that I am missing perhaps there is some thing with the driver that I am forgetting to kill between function calls. I am already using chrome driver headless so I don't know any other settigns for this.
I don't think the problem is from another part of the pyton because I did observe that multiple chrome instances getting started in the taskmanager as the program runs, that is the reason that throtles my cpu usage.
UPDATE 1: I have successfully implemented the the singleton. But now the problem is changed. When Singleton driver is called by the pytest Selenium itself is unable to connect. As of right now my main suspicion is pytest is somehow blocking itself through multi threading even though I have tried to implement lock mechanism to my singleton driver. Here is the Singleton Driver I have made.
class SdriverMeta(type):
_instances ={}
_lock: Lock = Lock()
def __call__(cls, *args, **kwargs):
with cls._lock:
if cls not in cls._instances:
instance = super().__call__(*args, **kwargs)
cls._instances[cls] = instance
return cls._instances[cls]
class SDriver:
_instance = None
def __new__(cls):
if cls._instance is None:
cls._instance = super(SDriver, cls).__new__(cls)
options = get_driver_options()
service = Service()
cls._instance.driver =webdriver.Chrome(service=service, options=options)
cls._instance.driver.implicitly_wait(30)
return cls._instance
def get_driver(self):
return self.driver
def close_driver(self):
if self.driver:
self.driver.close()
self.driver.quit()
SDriver._instance = None
And here is the problem. On one of the fixtures to create testing client group for proper certification testing, I try to get selenium to open my website but it gives me *MaxRetryError, Max retries exceeded with url: /session/2369e6d74474ac5c8ab85e652b356e0c/url (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x0000027B36908B50>: Failed to establish a new connection: [WinError 10061] *
@pytest.fixture(scope="function")
def create_test_group(self,request, setup_environment):
print("create test group")
params = request.param
groupmanager.create_testGroup(**params)# here we have the problem.
print("create test group yields")
yield
# Teardown steps if necessary (e.g., deleting the test group)
groupmanager.delete_testGroup()
def create_testGroup(NATBuddyPub=False, SSH=False, TCP=False, UDP=False):
'''
options = get_driver_options()
service = Service()
driver = webdriver.Chrome(service=service, options=options)
'''
sdriver=SDriver()
driver = sdriver.get_driver()
#driver.implicitly_wait(60)
website_url = default_config["website_url"]
driver.get(website_url) # this is where it actually messes up
# do some other stuff...
So so far my memory problem seems to be solved. But this time it is replaced by this maxRetryError.
UPDATE 2: This error was being caused by me not properly calling the class singleton's driver and instead calling self.
def get_driver(cls):
if cls._instance.driver is None:
options = get_driver_options()
service = Service()
cls._instance.driver = webdriver.Chrome(service=service, options=options)
cls._instance.driver.implicitly_wait(30)
return cls._instance.driver
this fixed the problem above. Yet I came right back at the square one. If I run the code line by line it seems to work fine. Yet once I run it, it overuses CPU and freezes my computer.
UPDATE 3: On @Techrookie89's comment I have added aditional cores to my machine and that seemed to work for the freezing and breaking part. The code had some sub-process making logic in some of the tests. I guess that was the problem.
I'm not an expert in python, but the problem that you state can be solved using the Singleton design pattern. You basically check whether the object is null or not, in case its null? you initialize it and when it's not you return the previously initialized instance.
It should be something similar to the snippet below:
class WebBrowser:
_instance = None
def __new__(cls):
if cls._instance is None:
cls._instance = super(WebBrowser, cls).__new__(cls)
options = get_driver_options()
service = Service()
driver = webdriver.Chrome(service=service, options=options)
driver.implicitly_wait(15)
cls._instance.driver = webdriver.Chrome()
return cls._instance
def get_driver(self):
return self.driver
def close_driver(self):
if self.driver:
self.driver.quit()
WebBrowser._instance = None
This can then be consumed in out tests as below:
browser = WebBrowser().get_driver()
browser.get("https://duckduckgo.com")
Note: If you want the type of webdriver to also be dynamic, I would suggest to read the Factory design pattern as well.