I think I have a silly question that I'm over thinking but looking for feedback. My team is using Selenium Java with Cucumber JUnit. I really feel like we aren't using it to its full capacity as far as execution goes, everything feels very outdated. We have our Scenarios in a variety of feature files based on functionality. A couple Scenarios for each feature are Tagged with @ Sanity for quick Sanity testing, and full Regression (hundreds of test cases) are tagged with either @ Regression1 or @ Regression2. We have a couple other tags we use but those are the primary ones.
When we run Regression, we have two Runner classes, TestRunner1.java with the cucumber options tag @ Regression1 and TestRunner2.java with tag @ Regression2. Then execution is done with the maven surefire plugin, is set to classes with 2 threads. So hundreds of test cases only executing two at time, kicked off in jenkins with a simple mvn test command on a vm. I'm newer to the team so I'm not trying to step on toes or rock the boat too hard but this seems really bad.
What is a better way to parallelize the execution in jenkins? Can obviously increase the number of TestRunner classes and split by class still, or switch to the method parameter of surefure. But I've found our vms really cant support more than like 4 threads at time without crashing/erroring out. Otherwise I was thinking like breaking down our Regression tags into @ Regression 1 thru 10, and in a Jenkins grid execution run 10 different mvn commands "mvn test -Dcucumber.filter.tags=@Regression1", "mvn test -Dcucumber.filter.tags=@Regression2", "mvn test -Dcucumber.filter.tags=@Regression3", etc.. with each getting assigned its own vm.
Thank you for any feedback before I dig more into this and present it to my team
I'm newer to the team so I'm not trying to step on toes or rock the boat too hard but this seems really bad.
Cucumber initially did not support parallel execution. So there were quite a few tools available to generate runner classes for individual scenarios. These could then be executed in parallel. These would presumably have been a better solution then tagging all tests by hand.
But none of that matters now. Since v4 of Cucumber parallel execution is supported natively.
But I've found our vms really cant support more than like 4 threads at time without crashing/erroring out.
It would be interesting to investigate why.
It could be that your VM can't handle more than 4 web drivers due to cpu/memory constraints. This cool be overcome by using larger instances and/ot allocating more resources.
It could also be that your tests expect certain elements to be present within a certain time frame (i.e. after Thread.sleep(10)) rather than explicitly wait for the element to be present.
Otherwise I was thinking like breaking down our Regression tags into @ Regression 1 thru 10, and in a Jenkins grid execution run 10 different mvn commands
This isn't a bad strategy.
Though rather than explicitly tagging tests consider using Cucumber with the JUnit Platform Launcher API.
With JUnit 5 you get a really nice programmatic API to discover, select and execute tests. You could have one job to discover all tests and write them to file. The next 10 jobs could then each pick up this file and execute every entry where entry_index modulo total_test_runners == test_runner_index.
As a starting point see Cucumber JUnit Platform Engine docs. Best to read it from top to bottom as well as the linker resources.