Search code examples
scalasbtscalatest

How to make SBT run test suites in parallel?


I have a bunch of integration tests running by sbt, given test N suites with 1..M tests per each suite. I have set fork in IntegrationTest := true, but test suites are always executed sequentially. According to the docs, this must not be the case: test suites should be executed concurrently.

the test suites are a class as following:

class MyTestSuite1 extends FlatSpec with Matchers 
...
it should "do A" {}
it should "do B" {}
class MyTestSuite2 extends FlatSpec with Matchers 
...
it should "do C" {}
it should "do D" {}

the problem

MyTestSuite1 and MyTestSuiteN are executed sequentially (by the alphabet order to be exact)

expectation

MyTestSuite1 and MyTestSuiteM are executed concurrently

env

.sbopts:

-J-Xms1G
-J-Xmx4G
-J-XX:MaxMetaspaceSize=512m
-J-Xss4M

note

I noticed that all test are running using the same pool and thread, for example, pool-1-thread-1 for all tests.

sbt version: 1.2.8 Scala: 2.12.8 os: MacOS 10.15, Ubuntu 19.04 Scalatest ver: 3.2.0-SNAP10

Tried sbt v. 1.3.2 - same result. Adding

testOptions in IntegrationTest += Tests.Argument(TestFrameworks.ScalaTest, "-P4"),

does not help.

============

Update

fork in(IntegrationTest, test) := true works on a global level, but I have 2 projects and I want to make it work to preserve relative path to the proj.

e.g.

lazy val `p1` = Project(id = "p1", base = file("./p1"))
  .configs(IntegrationTest)
  .settings(Defaults.itSettings: _*)
  .settings(
    fork in(IntegrationTest, test) := true,
    ...)

lazy val `p2` = Project(id = "p2", base = file("./p2"))
  .configs(IntegrationTest)
  .settings(Defaults.itSettings: _*)
  .settings(
    fork in(IntegrationTest, test) := true,
    ...)

does not run tests in parallel

instead, this runs in parallel, but, obviously, the home dir is set to be "." rather than to be "./p1" or "./p2" respectively:

fork in(IntegrationTest, test) := true

lazy val `p1` = Project(id = "p1", base = file("./p1"))
  .configs(IntegrationTest)
  .settings(Defaults.itSettings: _*)

Solution

  • SOLUTION:

    It appears there's testForkedParallel in IntegrationTest := true option which does exactly what I needed - it spawns new JVM per test suite.

    ==============

    Remark:

    So, the only problem is that now it spawns JVMs as many as the count of all available CPUs and I can't funnel only test concurrency:

    OPTION 1 - funnels all sbt processes to be only 4 in parallel

    concurrentRestrictions in Global := Seq(Tags.limitAll(4))
    

    OPTION 2 - just does nothing (test are in the subproject)

    concurrentRestrictions in Global += Tags.limit(Tags.Test, 4),