We run behave BDD tests in our pipeline. We run the tests in the docker container as part of the jenkins pipeline. Currently it takes ~10 minutes to run all the tests. We are adding a lot of tests and in few months, it might go upto 30 minutes. It is outputting a lot of information. I believe that if I reduce the amount of information it outputs, I can get the tests to run faster. Is there a way to control the amount of information behave outputs? I want to print the information only if something fails. I took a look at behave-parallel. Looks like it is in python 2.7. We are in python3. I was looking at various options behave provides.
behave -verbose=false folderName (I assumed that it will not output all the steps)
behave --logging-level=ERROR TQXYQ (I assumed it will print only if there is an error)
behave --logging-filter="Test Step" TQXYQ (I assumed it will print only the tests that has "Test Step" in it)
None of the above worked.
The current output looks like this
Scenario Outline: IsError is populated correctly based on Test Id -- @1.7 # TestName/Test.feature:187
Given the test file folder is set to /TestName/steps/ # common/common_steps.py:22 0.000s
And Service is running # common/common_steps.py:10 0.000s
Given request used is current.json # common/common_steps.py:26 0.000s
And request is modified to set X to q of type str # common/common_steps.py:111 0.000s
And request is modified to set Y to null of type str # common/common_steps.py:111 0.000s
And request is modified to set Z to USD of type str # common/common_steps.py:111 0.000s
When make a modification request # common/common_steps.py:37 0.203s
Then it returns 200 status code # common/common_steps.py:47 0.000s
And transformed result has IsError with 0 of type int # common/common_steps.py:92 0.000s
And transformed result has ErrorMessages contain [] # common/common_steps.py:52 0.000s
I want to print only all these things only if there is an error. If everything is passing, I don't want to display this information.
I think the default log level INFO will not impact the performance of your tests.
I am also using docker container to run the regression suite and it takes about 2 hours to run 2300 test scenarios. It took nearly a day before and here is what I did :
1. Run all test suite parallel.
This is the most important reason that will reduce the execution time.
We spent a lot of efforts to turn the regression suite to be parallel-able.
- make atomic, autonomous and independent tests so that you can run all your tests in parallel effectively.
- create a parallel runner to run tests on multiple processes. I am using multiprocessing and subprocessing libraries to do this.
I would not recommend behave-parallel because it is no longer active supported.
You can refer to this link :
http://blog.crevise.com/2018/02/executing-parallel-tests-using-behave.html?m=1
- using Docker Swarm to add more nodes into Selenium Grid.
You can scale up to add more nodes and the maximum numbers of nodes depend on the number of cpus. The best practice is number of node = number of cpu.
I have 4 PCs , each has 4 cores so I can scale up to 1 hub and 15 nodes.
2. Optimize synchronization in your framework.
Remove time.sleep()
Remove implicitly wait. Use explicitly wait instead.
Hope it helps.