I have a bunch of integration tests running by sbt, given test N suites with 1..M tests per each suite.
I have set fork in IntegrationTest := true
, but test suites are always executed sequentially. According to the docs, this must not be the case: test suites should be executed concurrently.
the test suites are a class as following:
class MyTestSuite1 extends FlatSpec with Matchers
...
it should "do A" {}
it should "do B" {}
class MyTestSuite2 extends FlatSpec with Matchers
...
it should "do C" {}
it should "do D" {}
MyTestSuite1 and MyTestSuiteN are executed sequentially (by the alphabet order to be exact)
MyTestSuite1 and MyTestSuiteM are executed concurrently
.sbopts:
-J-Xms1G
-J-Xmx4G
-J-XX:MaxMetaspaceSize=512m
-J-Xss4M
I noticed that all test are running using the same pool and thread, for example, pool-1-thread-1
for all tests.
sbt version: 1.2.8 Scala: 2.12.8 os: MacOS 10.15, Ubuntu 19.04 Scalatest ver: 3.2.0-SNAP10
Tried sbt v. 1.3.2 - same result. Adding
testOptions in IntegrationTest += Tests.Argument(TestFrameworks.ScalaTest, "-P4"),
does not help.
============
fork in(IntegrationTest, test) := true
works on a global level, but I have 2 projects and I want to make it work to preserve relative path to the proj.
e.g.
lazy val `p1` = Project(id = "p1", base = file("./p1"))
.configs(IntegrationTest)
.settings(Defaults.itSettings: _*)
.settings(
fork in(IntegrationTest, test) := true,
...)
lazy val `p2` = Project(id = "p2", base = file("./p2"))
.configs(IntegrationTest)
.settings(Defaults.itSettings: _*)
.settings(
fork in(IntegrationTest, test) := true,
...)
does not run tests in parallel
instead, this runs in parallel, but, obviously, the home dir is set to be "." rather than to be "./p1" or "./p2" respectively:
fork in(IntegrationTest, test) := true
lazy val `p1` = Project(id = "p1", base = file("./p1"))
.configs(IntegrationTest)
.settings(Defaults.itSettings: _*)
It appears there's testForkedParallel in IntegrationTest := true
option which does exactly what I needed - it spawns new JVM per test suite.
==============
So, the only problem is that now it spawns JVMs as many as the count of all available CPUs and I can't funnel only test concurrency:
OPTION 1 - funnels all sbt processes to be only 4 in parallel
concurrentRestrictions in Global := Seq(Tags.limitAll(4))
OPTION 2 - just does nothing (test are in the subproject)
concurrentRestrictions in Global += Tags.limit(Tags.Test, 4),