Forcing a build success even after test-failures in order to continue a build pipeline

Question

I have a series of JUnit 5 tests which must run in batches (Using tags). Once a batch finishes, the next batch should continue. If any tests fails in the first batch, the second batch does not start and this is not the behaviour I would like.

I've been told to use Gradle or Shell to return exit code zero (which is a success) so the batch of tests continue in a pipeline. I understand the implications of this - that the build will always show as passed.

---

In my build.gradle file I have tried:

test {
    ignoreFailures = true

    useJUnitPlatform {
        includeTags  itags
        excludeTags  etags
    }
    testLogging {
        events("passed", "skipped", "failed")
    }
    afterSuite {
        if (test.result == TestResult.FAILURE) {
            System.exit(0)
        }
    }
}

The afterSuite throws an error so I'm not sure I have the syntax error. The build doesn't recognise the "test."

I have also tried in Shell

set +x
gradle clean build test
if [ $? -ne 0 ]; then
  exit 0
fi
set -x

Which does not work either.

Answer 1

Below you can find a complete example project that hopefully does what you’re after (tested with Gradle 8.0.2). The project uses the JVM Test Suite Plugin to define one test suite for each batch of tests with a certain JUnit 5 tag.

Files in the Example Project

We have the following directory structure (not showing Gradle Wrapper files for brevity):

├── build.gradle
├── settings.gradle
└── src
    └── test
        └── java
            └── org
                └── example
                    └── MyTest.java

settings.gradle is empty and build.gradle has the following content:

plugins {
    id 'java'
}

repositories {
    mavenCentral()
}

testing {
    suites {
        configureEach {
            useJUnitJupiter('5.9.2')
            sources {
                java {
                    // Override the default which is based on the suite name.
                    srcDirs = ['src/test/java']
                }
            }
            targets {
                all {
                    testTask.configure {
                        ignoreFailures = true
                    }
                }
            }
        }

        // The "test" suite is provided by default. Let's use it for the first
        // batch tagged with "my-tag1".
        test {
            targets {
                all {
                    testTask.configure {
                        options {
                            includeTags 'my-tag1'
                        }
                    }
                }
            }
        }
        // The "test2" suite is created here. We use it for the second batch of
        // tests that are tagged with "my-tag2".
        test2(JvmTestSuite) {
            targets {
                all {
                    testTask.configure {
                        options {
                            includeTags 'my-tag2'
                        }
                    }
                }
            }
        }
    }
}

tasks.named('check').configure {
    dependsOn(testing.suites.names)
}

Here’s the content of MyTest.java:

package org.example;

import org.junit.jupiter.api.Tag;
import org.junit.jupiter.api.Test;

import static org.junit.jupiter.api.Assertions.fail;

public class MyTest {

    @Tag("my-tag1")
    @Test
    public void myTest1() {
        fail("test 1 fails");
    }

    @Tag("my-tag2")
    @Test
    public void myTest2() {
        fail("test 2 fails");
    }
}

Note that there are failing tests for all of our tags.

Running the Example Project

We can either run the batches individually using ./gradlew test and ./gradlew test2, or we can run them all in one go with ./gradlew check. For the latter, we get the following output:

> Task :test

MyTest > myTest1() FAILED
    org.opentest4j.AssertionFailedError at MyTest.java:13

1 test completed, 1 failed
There were failing tests. See the report at: file:///path/to/my_proj/build/reports/tests/test/index.html

> Task :test2

MyTest > myTest2() FAILED
    org.opentest4j.AssertionFailedError at MyTest.java:19

1 test completed, 1 failed
There were failing tests. See the report at: file:///path/to/my_proj/build/reports/tests/test2/index.html

BUILD SUCCESSFUL in 1s
4 actionable tasks: 4 executed

Despite the failing tests in each batch, the overall build is marked as successful (because of the used ignoreFailures = true configuration). The exit code of the overall build is hence also 0.