Search code examples
c++catch2

Catch2 runs test once more if all sections fail


I have the following piece of code that I run with the latest (2.4.0) version of Catch2:

#include "catch.hpp"
#include <iostream>
TEST_CASE("Test") {
    int x = 0;
    SECTION("A") {
            std::cout << "A";
            ++x;
            REQUIRE(x == 1);
    }
    SECTION("B") {
            std::cout << "B";
            ++x;
            REQUIRE(x == 1);
    }
    std::cout << "X\n";
    REQUIRE(x == 1);
}

If I run this, everything works as expected and I get:

AX
BX
=================================================================
All tests passed (4 assertions in 1 test case)

Clearly, the test case is run twice, once for each section.

If I change the assert in one of the sections, say to REQUIRE(x == 0), again, everything works as expected, Catch2 runs each section once and tells me the first one failed. However, if I change the assert to REQUIRE(x == 0) in both sections, the result is confusing (I have shortened it slightly):

A
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
a.out is a Catch v2.4.0 host application.
Run with -? for options
---------------------------------------------------------------------
Test
  A
---------------------------------------------------------------------
test.cpp:10: FAILED:
  REQUIRE( x == 0 )
with expansion:
  1 == 0
B--------------------------------------------------------------------
Test
  B
---------------------------------------------------------------------
test.cpp:15: FAILED:
  REQUIRE( x == 0 )
with expansion:
  1 == 0

X
---------------------------------------------------------------------
Test
---------------------------------------------------------------------
test.cpp:19: FAILED:
  REQUIRE( x == 1 )
with expansion:
  0 == 1
=====================================================================
test cases: 1 | 1 failed
assertions: 3 | 3 failed

The test is clearly run three times, the last time bypassing both of the sections. Is this behaviour expected? I have tried looking at the Catch2 docs but could not find anything relevant.


Solution

  • From the docs:

    One leaf section is executed on each run through a TEST_CASE. The other sections are skipped. Next time through the next section is executed, and so on until no new sections are encountered.

    So the problematic test has to be run 3 times.