How to skip a failed test

In go, are you allowed skip a test that has already failed?

Context:

I have a heisenbug which I cannot currently determine the cause of. It causes some tests to sometimes fail. By examining various logs I can identify the failure mode. I would like to do something like:

if t.Failed() {
    if strings.Contains(string(suite.Stdout), "connection reset by peer") {
        t.Skip("Skip Test failed ")
    }
}

The tests are valuable enough that I want to run them in CI despite the heisenbug so this is just a temporary workaround.

This doesn't work. Is there a way to retrospectively skip a test if it fails?

Solution

The short answer is no. You can skip a test or fail it but not both.

The designers of go consider trying to skip a test as trying to subvert the test framework so you should not try to do this:

See for example https://github.com/golang/go/issues/16502

This is documented but its easy to miss:

If a test fails (see Error, Errorf, Fail) and is then skipped, it is still considered to have failed.

If you have a reliable way of detecting the heisenbug you should run it before making any test assertions. So rather than:

// execute
executeThingBeingTested()

// verify
assert.Equal(t, expected, actual)

// recover if needed
if t.Failed() {
    // detect heisenbug
    if strings.Contains(string(suite.Stdout), "connection reset by peer") {
        t.Skip("Skip Test failed ")
    }
}

You should instead structure your tests like:

// execute
executeThingBeingTested()

// skip if needed
if strings.Contains(string(suite.Stdout), "connection reset by peer") {
    t.Skip("Skip Test failed ")
}

// verify
assert.Equal(t, expected, actual)

This means that you cannot alternate between multiple execution and verification stages in a single test but its good practice to have only a single execution and verification stage in each test anyway. i.e. four phase testing

Now if you really really want to do it you can go low-level. This is probably not a good idea but included for completeness. Peering down the rabbit hole may help show you don't want to go there. This takes in consideration this question and how the testing package is implemented

    t := suite.T()

    // low-level hackery - undo the failed state so we can skip a test
    pointerVal := reflect.ValueOf(t)
    val := reflect.Indirect(pointerVal)
    member := val.FieldByName("failed")
    ptrToFailedFlag := unsafe.Pointer(member.UnsafeAddr())
    realPtrToFailedFlag := (*bool)(ptrToFailedFlag)
    *realPtrToFailedFlag = false

If this level of hackery is not enough to convince you what a bad idea this is you might want to note the implementation of fail() at the time of writing:

        // Fail marks the function as having failed but continues execution.
   605  func (c *common) Fail() {
   606      if c.parent != nil {
   607          c.parent.Fail()
   608      }
   609      c.mu.Lock()
   610      defer c.mu.Unlock()
   611      // c.done needs to be locked to synchronize checks to c.done in parent tests.
   612      if c.done {
   613          panic("Fail in goroutine after " + c.name + " has completed")
   614      }
   615      c.failed = true
   616  }

You can see that as soon as Fail() is invoked any parent tests are also marked as having failed. So if you are using something like testify/suite to organise tests into suites, to unfail a test you would also have to unfail the parent test but if and only if no other tests in the suite have failed. So altering the testing() package to allow skip to occur after fail interacts poorly with the idea of nested tests.