Search code examples
jenkinscucumbermaven-surefire-pluginallure

Set Jenkins build to stable using Allure/surefire-plugin & cucumber when 95% of the tests passes


The build is turned yellow even if one test has failed and that annoying. I want to build to turn to unstable only if lass than 95% of the tests has failed.

Is there a way to force the jenkins build to be stable when using Allure/surefire-plugin & cucumber?

enter image description here

[INFO] 
[INFO] Results:
[INFO] 
[WARNING] Flakes: 
[WARNING] inloggen.Controlle verplichte velden inlogpagina
[INFO]   Run 1: PASS
[INFO]   Run 2: PASS
[ERROR]   Run 3: Kan de tekst "Er is een authenticatie fout opgetreden" niet vinden!
[ERROR]   Run 4: Kan de tekst "Er is een authenticatie fout opgetreden" niet vinden!
[INFO] 
[INFO] 
[WARNING] Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Flakes: 1
[INFO] 
[INFO] 
[INFO] --- maven-failsafe-plugin:3.0.0-M5:verify (default) @ XXX---
[INFO] 
[INFO] --- maven-jar-plugin:2.4:jar (default-jar) @ XXX_cucumber_integratietests ---
[WARNING] JAR will be empty - no content was marked for inclusion!
[INFO] Building jar: C:\jenkins\workspace\XXX_CucumberIntegrationTestPipe\clone\test\functional\cucumber\XXX_cucumber_integratietests\target\XXX_cucumber_integratietests-1.2-SNAPSHOT.jar
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:12 min (Wall Clock)
[INFO] Finished at: 2020-09-14T15:33:44+02:00
[INFO] ------------------------------------------------------------------------
[XXX_CucumberIntegrationTestPipe] $ C:\ws\allure-2.13.5\bin\allure.bat generate C:\jenkins\workspace\XXX_CucumberIntegrationTestPipe\clone\test\functional\cucumber\XXX_cucumber_integratietests\target\allure-results -c -o C:\jenkins\workspace\XXXCucumberIntegrationTestPipe\clone\test\functional\cucumber\target\allure-report
Report successfully generated to C:\jenkins\workspace\XXX_CucumberIntegrationTestPipe\clone\test\functional\cucumber\target\allure-report
Allure report was successfully generated.
Creating artifact for the build.
Artifact was added to the build.
Build step 'Allure Report' changed build result to UNSTABLE
Finished: UNSTABLE

Solution

  • Setting the threadshold to 95% is not possible in Maven. The Maven does not have such ambition. It can be configured only in Jenkins CI.