Is there a way to run a test, but not mark the whole suite as failed if it fails?

Question

Using Scalatest is there a way to mark a test as "optional". What I mean is running it anyway, but if it fails do not mark the whole test execution as failed (Return 0 instead of 1 to the shell).

I am aware that ignore exists as a replacement for in, but that does not exactly do what I am willing to do.

The reason I want to do this is that I have a project in which some tests might run in some networks and in some other fail.

Is this possible?

Answer 1

You can use tags to define groups of tests. You could either whitelist or blacklist tests. If you want to whitelist tests, define a tag for each environment, and if a test should pass in that environment, give it that tag. Then, in each environment, include tests with the tag for that environment with -n. If you want to blacklist tests, tag tests that should fail in each environment, and then exclude tests with that tag using -l. Filtering tests is done with runner configuration.

Here is an example for blacklisting tests that require an external service.

object ServiceTest extends Tag("com.mycompany.tags.ServiceTest")

class MyTest extends WordSpec {
  "a foo" should {
    "connect to the service" taggedAs ServiceTest in {
      // ...
    }

    "do something without the service" in {
      // ...
    }
  }
}

You could also use cancel() to abort a test. I'm not sure what you mean by "aborts everything", only the current test case is canceled. But I think it is better to set expectations for tests to pass or fail in a given environment, and if they don't, investigate. Otherwise, it is too easy to ignore canceled tests if the network is unavailable and there is a real problem, say the url changed.

class MyTest extends WordSpec {
  "a foo" should {
    "connect to the service" in {
      if (serviceUnavailable()) cancel("Can't connect to service.")

      // ...
    }

    "do something without the network" in {
      // ...
    }
  }

  def serviceUnavailable(): Boolean = ???
}