SBT don't run any Specs2 test in custom test configuration

Question

I have a multiproject with some settings and custom configs shared between them. Those settings are configs that should run Specs2 tests by tag. However, when I run those new configs they aren't running any test at all.

import sbt.TestFrameworks.Specs2
import sbt.Tests.Argument
import sbt._
import sbt.Keys._

trait Settings extends Dependencies {

  lazy val FunctionalTest = config("functional") extend(Test)

  val commonConfigs = Seq(FunctionalTest)

  val commonSettings =
    inConfig(FunctionalTest)(Defaults.testSettings) ++
    Seq(
      libraryDependencies ++= mainDeps,
      libraryDependencies ++= testDeps map (_ % "test"),

      testOptions in Test += Argument(Specs2, "exclude", TestTag.FunctionalTest),
      testOptions in FunctionalTest += Argument(Specs2, "include", TestTag.FunctionalTest)
    )
}

After defining functional:test task like this it doesn't run any tests while test task would run all tests in all subprojects which aren't tagged with TestTag.FunctionalTest.

Removing testOptions in FunctionalTest += Argument(Specs2, "include", TestTag.FunctionalTest) doesn't change the result, but removing inConfig(FunctionalTest)(Defaults.testSettings) ++ make functional:test behave exactly like test (run all except TestTag.FunctionaTest).

How can I make test run everything except TestTag.FunctionalTest tagged tests (as it is now) and functional:test to run only tagged tests? I looked at SBT and Spec2 documentation, but they only have working examples for filtering tests by their name (SBT) or using command line arguments to filter them (Specs2) and nowhere could I find information on how to combine them.

Answer 1

There were couple of things that I had to set in order for everything to work as I wanted it to. Final solution looks something like this:

trait Settings extends Dependencies {

  lazy val FunctionalTest = config("functional") extend(Test)

  val commonConfigs = Seq(FunctionalTest)

  val functionalSettings =
    inConfig(FunctionalTest)(Defaults.testTasks) ++
    Seq(testOptions in FunctionalTest := Argument(Specs2, "include", TestTag.FunctionalTest))

  val commonSettings =
     ++
    Seq(
      libraryDependencies ++= mainDeps,
      libraryDependencies ++= testDeps map (_ % "test"),

      testOptions in Test += Argument(Specs2, "exclude", TestTag.FunctionalTest)
    )
}

used as:

project.
  configs(commonConfigs).
  settings(functionalSettings: _*).
  settings(commonSettings: _*)

Things that I had to fix in order to make it work as I intended:

1. splitting settings into several steps:

   When I created a large batch of settings by concatenating them and applying project.settings(allSettingsAtOnce: _*) things would break. Separating them was the first step towards making functional:test works. It also allowed me to apply fix #3 since testOptions := option is not available when you are creating a sequence of settings.

2. replacing Defaults.testSettings with Defaults.testTasks.

   With inConfig(FunctionalTest)(Defaults.testSettings) no tests were run. Task simply succeeded right after start without any report from test framework.

3. replacing testOptions in FunctionalTest += option with testOptions in FunctionalTest := options.

   With += I ended up with both exclude functional and include functional test framework arguments at once. I guess when I append argument to Test it is automatically propagated to everything that extends it, and so I have two conflicting arguments in derived config. With := I get rid of unwanted argument and things works again.

None of those quirks I found described in docs, solution is effect of putting together solutions of several questions I found and a lot of tries.