Search code examples
c++catch2

Running unit tests excluding a specific tag in Catch2

Can I run test cases based on "not matching" a particular tag in Catch2?

TEST_CASE("Check the data validity","[Working]"){
  REQUIRE(true);
}

TEST_CASE("Check the input","[InProgress]"){
  REQUIRE(true);
}
TEST_CASE("Validate the response","[NotWorking]"){
  REQUIRE(false);
}

I want to invoke testcases that are not falling in [NotWorking] tag until I finish implementing that functionality.

Solution

  • In your case add to the command line:

    ~[NotWorking]

    Source: https://github.com/catchorg/Catch2/blob/master/docs/command-line.md#specifying-which-tests-to-run

    Specifying which tests to run

    <test-spec> ...

    By providing a test spec, you filter which tests will be run. If you call Catch2 without any test spec, then it will run all non-hidden test cases. A test case is hidden if it has the [!benchmark] tag, any tag with a dot at the start, e.g. [.] or [.foo].

    There are three basic test specs that can then be combined into more complex specs:

    • Full test name, e.g. "Test 1".

    This allows only test cases whose name is "Test 1".

    • Wildcarded test name, e.g. "*Test", or "Test*", or "*Test*".

    This allows any test case whose name ends with, starts with, or contains in the middle the string "Test". Note that the wildcard can only be at the start or end.

    • Tag name, e.g. [some-tag].

    This allows any test case tagged with "[some-tag]". Remember that some tags are special, e.g. those that start with "." or with "!".

    You can also combine the basic test specs to create more complex test specs. You can

    • Concatenate specs to apply all of them, e.g. [some-tag][other-tag].

    This allows test cases that are tagged with both "[some-tag]" and "[other-tag]". A test case with just "[some-tag]" will not pass the filter, nor will test case with just "[other-tag]".

    • Comma-join specs to apply any of them, e.g. [some-tag],[other-tag].

    This allows test cases that are tagged with either "[some-tag]" or "[other-tag]". A test case with both will obviously also pass the filter.

    Note that commas take precendence over simple concatenation. This means that [a][b],[c] accepts tests that are tagged with either both "[a]" and "[b]", or tests that are tagged with just "[c]".

    • Negate the spec by prepending it with ~, e.g. ~[some-tag].

    This rejects any test case that is tagged with "[some-tag]". Note that rejection takes precedence over other filters.

    Note that negations always binds to the following basic test spec. This means that ~[foo][bar] negates only the "[foo]" tag and not the "[bar]" tag.

    Note that when Catch2 is deciding whether to include a test, first it checks whether the test matches any negative filters. If it does, the test is rejected. After that, the behaviour depends on whether there are positive filters as well. If there are no positive filters, all remaining non-hidden tests are included. If there are positive filters, only tests that match the positive filters are included.

    You can also match test names with special characters by escaping them with a backslash ("\"), e.g. a test named "Do A, then B" is matched by "Do A, then B" test spec. Backslash also escapes itself.

    Examples

    Given these TEST_CASEs,

    TEST_CASE("Test 1") {}

TEST_CASE("Test 2", "[.foo]") {}  

TEST_CASE("Test 3", "[.bar]") {}

TEST_CASE("Test 4", "[.][foo][bar]") {}

    this is the result of these filters

    ./tests                      # Selects only the first test, others are hidden 
./tests "Test 1"             # Selects only the first test, other do not match 
./tests ~"Test 1"            # Selects no tests. Test 1 is rejected, other tests are hidden 
./tests "Test *"             # Selects all tests. 
./tests [bar]                # Selects tests 3 and 4. Other tests are not tagged [bar] 
./tests ~[foo]               # Selects test 1, because it is the only non-hidden test without [foo] tag 
./tests [foo][bar]           # Selects test 4. 
./tests [foo],[bar]          # Selects tests 2, 3, 4.
./tests ~[foo][bar]          # Selects test 3. 2 and 4 are rejected due to having [foo] tag 
./tests ~"Test 2"[foo]       # Selects test 4, because test 2 is explicitly rejected 
./tests [foo][bar],"Test 1"  # Selects tests 1 and 4. 
./tests "Test 1*"            # Selects test 1, wildcard can match zero characters

    Note: Using plain asterisk on a command line can cause issues with shell expansion. Make sure that the asterisk is passed to Catch2 and is not interpreted by the shell.