I'm trying to achieve parallelism in a new thread safe test framework I'm writing. I have a DataProviderManager.java class which contains my static DataProvider
@DataProvider(parallel = true, name = Config.StaticProps.DEFAULT_WEB_DATA_PROVIDER)
public static Object[][] defaultWebDataProvider() {
return new Object[][] {
new Object[]{"chrome", "69", "MAC 10.12"},
new Object[]{"safari", "11.0", "Mac 10.13"},
new Object[]{"safari", "11.0", "Mac 10.13"}
new Object[]{"chrome", "70", "Mac 10.13"}
new Object[]{"firefox", "63", "MAC 10.12"},
new Object[]{"firefox", "62", "MAC 10.12"}
};
}
I have a test class that passes 3 parameters from my DataProvider to ThreadLocal getters and setters to set Selenium Capabilities per thread instantiating a driver instance with these capabilities. The test itself is simply opening a specific url based on some runtime params.
@Factory(dataProvider = StaticProps.DEFAULT_WEB_DATA_PROVIDER, dataProviderClass = DataProviderManager.class)
//@Factory(dataProvider = StaticProps.DEFAULT_WEB_DATA_PROVIDER)
public WebExampleTest(String browser, String browserVersion, String platform) {
super.setRunParams(browser, browserVersion, platform);
}
@BeforeMethod(alwaysRun = true)
public void setupTest() {
home = new Home();
}
@TestCaseId("")
@Features({GroupProps.WEB})
@Test(groups = {GroupProps.DEBUG}, dataProvider = StaticProps.DEFAULT_WEB_DATA_PROVIDER, dataProviderClass = DataProviderManager.class)
@Parameters({ ParamProps.WEB, ParamProps.MOBILE_WEB })
public void openSiteTest(String browser, String browserVersion, String platform) {
Logger.logMessage("Current Thread ID: " + Thread.currentThread().getId());
new WebInteract(null, null).pause(1000).openUrl(URLBuilder.buildUrl());
}
Here is my testing.xml file
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="Test Engineering Core Suite"
parallel="true"
data-provider-thread-count="6"
thread-count="6"
verbose="2">
<!-- LISTENERS -->
<listeners>
<listener class-name="com.msg.test.core.listeners.TestListeners" />
<listener class-name="com.msg.test.core.listeners.SuiteListeners" />
<listener class-name="com.msg.test.core.listeners.TestAnnotationTransformer" />
</listeners>
<!-- TEST EXECUTION QUEUE -->
<test verbose="1" name="All Tests" annotations="JDK">
<!-- TEST GROUPS
"Full" - all tests in the test suite.
"Broken" - those tests that are broken and safe to ignore until they can be resolved.
-->
<!--<groups>-->
<!--<run>-->
<!--<include name="Debug" />-->
<!--<exclude name="Broken" />-->
<!--</run>-->
<!--</groups>-->
</test>
</suite>
Here is part of the output to show you whats currently happening
2018-11-18 12:49:37.633 ========NEW TEST SESSION========
2018-11-18 12:49:37.633 Desktop OS: Mac 10.13
2018-11-18 12:49:37.633 Browser: chrome
2018-11-18 12:49:37.633 Browser Version: 70
2018-11-18 12:49:37.634 Current Thread ID: 66
2018-11-18 12:49:37.635 Pause for '1000' milliseconds.
Nov 18, 2018 12:49:37 AM org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: OSS
2018-11-18 12:49:37.846 Session ID: 622fda273344469ab98aef0d06f90315
2018-11-18 12:49:37.847 ========NEW TEST SESSION========
2018-11-18 12:49:37.847 Desktop OS: Mac 10.13
2018-11-18 12:49:37.847 Browser: chrome
2018-11-18 12:49:37.848 Browser Version: 70
2018-11-18 12:49:37.849 Current Thread ID: 68
2018-11-18 12:49:37.849 Pause for '1000' milliseconds.
2018-11-18 12:49:37.950 Open url 'https://www.example.com'.
2018-11-18 12:49:38.571 Open url 'https://www.example.com'.
2018-11-18 12:49:38.571 Open url 'https://www.example.com'.
2018-11-18 12:49:38.576 ======SUCCESS======
2018-11-18 12:49:38.577 Desktop OS: Mac 10.13
2018-11-18 12:49:38.577 Browser: chrome
2018-11-18 12:49:38.577 Browser Version: 70
2018-11-18 12:49:38.640 Open url 'https://www.example.com'.
2018-11-18 12:49:38.854 Open url 'https://www.example.com'.
2018-11-18 12:49:39.059 ======SUCCESS======
2018-11-18 12:49:39.060 Desktop OS: Mac 10.13
2018-11-18 12:49:39.060 Browser: chrome
2018-11-18 12:49:39.060 Browser Version: 70
2018-11-18 12:49:39.083 ======SUCCESS======
2018-11-18 12:49:39.084 Desktop OS: Mac 10.13
2018-11-18 12:49:39.084 Browser: chrome
2018-11-18 12:49:39.084 Browser Version: 70
2018-11-18 12:49:39.126 ======SUCCESS======
2018-11-18 12:49:39.126 Browser: chrome
2018-11-18 12:49:39.126 Browser Version: 70
2018-11-18 12:49:39.393 ======SUCCESS======
as you can see I am able to run the same test in parallel on different threads but each thread is running on the same browser, version and platform from my DataProvider.
My question is instead of running multiple threads each executing the same data (browser, version platform - chrome, 70, Mac 10.13) from DataProvider at the same time, how do run I run each (browser, version, platform)
new Object[]{"chrome", "69", "MAC 10.12"}, **This should be thread 1**
new Object[]{"safari", "11.0", "Mac 10.13"}, **This should be thread 2**
new Object[]{"safari", "11.0", "Mac 10.13"} **This should be thread 3**
new Object[]{"chrome", "70", "Mac 10.13"} **This should be thread 4**
new Object[]{"firefox", "63", "MAC 10.12"}, **This should be thread 5**
new Object[]{"firefox", "62", "MAC 10.12"} **This should be thread 6**
from DataProvider on different threads at the same time?
Here's how TestNG works
A regular @Test method coupled with a data provider
When a normal @Test annotated test method needs to be run multiple times but with different sets of data, you couple it with a data provider. To run it in parallel, you basically enable the parallel attribute on the @DataProvider annotation and specify the attribute parallel=methods at the <suite> tag level.
Below is a sample that shows all of this in action. It makes use of an org.testng.IHookable interface so that one can basically extract out the parameters that are being sent to the @Test method via the data provider and use them to do any setup that is required for every iteration.
You could use a @BeforeMethod annotation as well, but its not possible to extract out the parameters being sent to the @Test method by the data provider, from within a @BeforeMethod.
import org.testng.IHookCallBack;
import org.testng.IHookable;
import org.testng.ITestResult;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;
public class TestClassWithDataDrivenTest implements IHookable {
@Override
public void run(IHookCallBack callBack, ITestResult result) {
beforeMethod(callBack.getParameters());
callBack.runTestMethod(result);
}
public void beforeMethod(Object[] parameters) {
String browser = parameters[0].toString();
String browserVersion = parameters[1].toString();
String platform = parameters[2].toString();
String msg =
String.format(
"[beforeMethod()-Thread Id : %d] [%s] flavor version [%s] on platform [%s]",
Thread.currentThread().getId(), browser, browserVersion, platform);
System.err.println(msg);
}
@Test(dataProvider = "dp")
public void testMethod(String browser, String browserVersion, String platform) {
String msg =
String.format(
"[testMethod()-Thread Id : %d] [%s] flavor version [%s] on platform [%s]",
Thread.currentThread().getId(), browser, browserVersion, platform);
System.err.println(msg);
}
@DataProvider(parallel = true, name = "dp")
public static Object[][] defaultWebDataProvider() {
return new Object[][] {
new Object[] {"chrome", "69", "MAC 10.12"},
new Object[] {"safari", "11.0", "Mac 10.13"},
new Object[] {"safari", "11.0", "Mac 10.13"},
new Object[] {"chrome", "70", "Mac 10.13"},
new Object[] {"firefox", "63", "MAC 10.12"},
new Object[] {"firefox", "62", "MAC 10.12"}
};
}
}
The corresponding output when you run this test is as below
[beforeMethod()-Thread Id : 11] [chrome] flavor version [69] on platform [MAC 10.12]
[beforeMethod()-Thread Id : 12] [safari] flavor version [11.0] on platform [Mac 10.13]
[beforeMethod()-Thread Id : 14] [chrome] flavor version [70] on platform [Mac 10.13]
[beforeMethod()-Thread Id : 13] [safari] flavor version [11.0] on platform [Mac 10.13]
[beforeMethod()-Thread Id : 15] [firefox] flavor version [63] on platform [MAC 10.12]
[testMethod()-Thread Id : 12] [safari] flavor version [11.0] on platform [Mac 10.13]
[beforeMethod()-Thread Id : 16] [firefox] flavor version [62] on platform [MAC 10.12]
[testMethod()-Thread Id : 13] [safari] flavor version [11.0] on platform [Mac 10.13]
[testMethod()-Thread Id : 14] [chrome] flavor version [70] on platform [Mac 10.13]
[testMethod()-Thread Id : 11] [chrome] flavor version [69] on platform [MAC 10.12]
[testMethod()-Thread Id : 16] [firefox] flavor version [62] on platform [MAC 10.12]
[testMethod()-Thread Id : 15] [firefox] flavor version [63] on platform [MAC 10.12]
===============================================
Default Suite
Total tests run: 6, Passes: 6, Failures: 0, Skips: 0
===============================================
A @Factory coupled with a data provider
One uses a @Factory method (this annotation could be added on top of a constructor or on a static method which returns an Object[]) when one would like to produce test class instances.
A test class instance is defined as a regular class that houses one or more @Test methods in it. The factory when coupled with a data provider, uses the data provided by the data provider to instantiate each of the test class.
If one would like to have TestNG run the @Test methods in all of the instances in parallel at one shot (TestNG would still run all the @Test methods in each of the test class in sequential order), then you can specify it via the attribute parallel=instances on the <suite> tag.
Please note that you cannot run all the @Test methods in each of the test class instances in parallel. TestNG AFAIK doesn't have an option of enabling parallelism at that level.
Below is a sample that shows this in action
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Factory;
import org.testng.annotations.Test;
public class TestClassWithFactory {
private String browser, browserVersion, platform;
@Factory(dataProvider = "dp")
public TestClassWithFactory(String browser, String browserVersion, String platform) {
this.browser = browser;
this.browserVersion = browserVersion;
this.platform = platform;
}
@BeforeMethod
public void beforeMethod() {
String msg =
String.format(
"[Instance id %d][beforeMethod()-Thread Id : %d] [%s] flavor version [%s] on platform [%s]",
this.hashCode(), Thread.currentThread().getId(), browser, browserVersion, platform);
System.err.println(msg);
}
@Test
public void testMethod() {
String msg =
String.format(
"[Instance id %d][testMethod()-Thread Id : %d] [%s] flavor version [%s] on platform [%s]",
this.hashCode(), Thread.currentThread().getId(), browser, browserVersion, platform);
System.err.println(msg);
}
@DataProvider(parallel = true, name = "dp")
public static Object[][] defaultWebDataProvider() {
return new Object[][] {
new Object[] {"chrome", "69", "MAC 10.12"},
new Object[] {"safari", "11.0", "Mac 10.13"},
new Object[] {"safari", "11.0", "Mac 10.13"},
new Object[] {"chrome", "70", "Mac 10.13"},
new Object[] {"firefox", "63", "MAC 10.12"},
new Object[] {"firefox", "62", "MAC 10.12"}
};
}
}
The testng suite xml file is as below
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="53345934_suite" parallel="instances" verbose="2">
<test name="53345934_test">
<classes>
<class name="com.rationaleemotions.stackoverflow.qn53345934.TestClassWithFactory"/>
</classes>
</test>
</suite>
Execution output is as below:
...
... TestNG 7.0.0-beta1 by Cédric Beust ([email protected])
...
[Instance id 1627821297][beforeMethod()-Thread Id : 15] [safari] flavor version [11.0] on platform [Mac 10.13]
[Instance id 1549409129][beforeMethod()-Thread Id : 14] [chrome] flavor version [70] on platform [Mac 10.13]
[Instance id 922872566][beforeMethod()-Thread Id : 11] [firefox] flavor version [63] on platform [MAC 10.12]
[Instance id 1132547352][beforeMethod()-Thread Id : 12] [safari] flavor version [11.0] on platform [Mac 10.13]
[Instance id 1525037790][beforeMethod()-Thread Id : 13] [chrome] flavor version [69] on platform [MAC 10.12]
[Instance id 1132547352][testMethod()-Thread Id : 12] [safari] flavor version [11.0] on platform [Mac 10.13]
[Instance id 1549409129][testMethod()-Thread Id : 14] [chrome] flavor version [70] on platform [Mac 10.13]
[Instance id 1525037790][testMethod()-Thread Id : 13] [chrome] flavor version [69] on platform [MAC 10.12]
[Instance id 922872566][testMethod()-Thread Id : 11] [firefox] flavor version [63] on platform [MAC 10.12]
[Instance id 1627821297][testMethod()-Thread Id : 15] [safari] flavor version [11.0] on platform [Mac 10.13]
[Instance id 1651855867][beforeMethod()-Thread Id : 12] [firefox] flavor version [62] on platform [MAC 10.12]
[Instance id 1651855867][testMethod()-Thread Id : 12] [firefox] flavor version [62] on platform [MAC 10.12]
PASSED: testMethod
PASSED: testMethod
PASSED: testMethod
PASSED: testMethod
PASSED: testMethod
PASSED: testMethod
===============================================
53345934_test
Tests run: 6, Failures: 0, Skips: 0
===============================================
===============================================
53345934_suite
Total tests run: 6, Passes: 6, Failures: 0, Skips: 0
===============================================