I have jenkins pipeline with some stages and I have implemented post block to rerun the test if the previous build was failed. That part is working fine, the issue is the build is still marked as failed at the end and also it is skipping couple stages where it generates html report and send email notifications.
def secrets =[[path: /platforms/testing',engineVersion:1,secretValues:[
[envVar:'TEST_USERNAME',vaultKey:'username-appauth'],
[envVar:'TEST_PASSWORD',vaultKey:'password-appauth']
]]
]
def configuration =[vaultUrl:'https://vault.com',vaultCredentialId:'0e561836be49',engineVersion:1]
pipeline {
agent {
node {
label 'test'
}
}
environment {
PATH = "${tool 'gradle683'}/bin:${env.PATH}"
JAVA_HOME = "/var/lib/jenkins/tools/hudson.model.JDK/openjdk-11/jdk-11.0.19+7"
dartsURL = "https://dartts-webapp-test.testapp1.com/"
// hubURL = "http://ipaddress.irmnet.ds2.com:4444/wd/hub"
hubURL = "https://selenium-grid.test-eks-nonprod.com/wd/hub"
}
stages {
stage('Get Git Repo') {
steps {
script {
git branch:'development', credentialsId:'GitHub-test', url:
'https://github.com/test/test-Webapp-Selenium-Automation'
}
}
}
stage('Gradle Build') {
tools {
jdk 'openjdk-11'
}
environment {
JAVA_HOME = "${jdk}"
PATH = "${JAVA_HOME}/bin:${env.PATH}"
}
steps {
script {
sh 'chmod +x ./gradlew'
}
}
}
stage('Run Dartts Tests') {
steps {
script {
withVault([configuration:configuration, vaultSecrets:secrets]){
env.ADFSUsername="InvalidUsername"
env.ADFSPassword = env.TEST_PASSWORD
}
catchError(buildResult:'FAILURE', stageResult:'FAILURE'){
sh "./gradlew clean runDarttsLoginCheck"
}
}
}
post {
failure {
script {
withVault([configuration:configuration, vaultSecrets:secrets]){
env.ADFSUsername = env.TEST_USERNAME
env.ADFSPassword = env.TEST_PASSWORD
}
catchError(buildResult:'FAILURE', stageResult:'FAILURE'){
sh "./gradlew clean runDarttsLoginCheck"
}
}
}
}
}
stage('Generate HTML report') {
steps {
cucumber buildStatus:'',
reportTitle:'Dartts report',
fileIncludePattern:'**/*.json',
trendsLimit:10,
classifications:[
[
'key':'Browser',
'value':'Chrome'
]
]
}
}
stage('Checking log') {
steps {
script {
//List
<String> log= currentBuild.rawBuild.getLog(100)
boolean Pass = currentBuild.rawBuild.getLog(100).contains("Finished: SUCCESS")
boolean Fail= currentBuild.rawBuild.getLog(100).contains("ERROR: script returned exit code 1")
boolean Exception = currentBuild.rawBuild.getLog(100).contains("FAILURE: Build failed with an exception.")
boolean CompileError=currentBuild.rawBuild.getLog(100).contains("Execution failed for task ':compileJava'.")
if (Pass) {
echo 'Passed'
} else if (Fail | Exception) {
if (!CompileError) {
echo "Failed."
emailext(
subject:"DARTTS test Failed",
body:"Automation tests for DARTTS are failed. Please click the link to access the test results for more details \n\n Cucumber report\n" + env.BUILD_URL + "cucumber-html-reports/overview-features.html",
to:'vtang@associates.com',
from:"donotreply@cloudbees.com")
} else {
echo "Compile Error."
emailext(
subject:"DARTTS test Health Check Failure in PROD due to Java Compile error",
body:"Automation tests Failed due to Java compile error. Please click the link to access the test results for more details \n\n Cucumber report\n" + env.BUILD_URL + "/cucumber-html-reports/overview-features.html",
to:'vtang@associates.com'
from:"donotreply@cloudbees.com")
}
}
}
}
}
}
}
Console output for the error
1 test completed, 1 failed
> Task :runDarttsLoginCheck FAILED
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':runDarttsLoginCheck'.
> There were failing tests. See the report at: file:///var/lib/jenkins/workspace/Automation%20Jobs/Prod-Smoke-Login-Jobs/Dartts-Webapp-Smoke-Login-Pipeline/build/reports/tests/runDarttsLoginCheck/index.html
* Try:
> Run with --scan to get full insights.
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.2/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD FAILED in 1m 18s
6 actionable tasks: 6 executed
[Pipeline] }
[Pipeline] // script
Post stage
[Pipeline] script
[Pipeline] {
[Pipeline] withVault
Retrieving secret: secret/raven/cloudbees/platforms/testing
[Pipeline] {
[Pipeline] }
[Pipeline] // withVault
[Pipeline] catchError
[Pipeline] {
[Pipeline] sh
+ ./gradlew clean runDarttsLoginCheck
> Task :clean
> Task :compileJava
> Task :processResources
> Task :classes
> Task :compileTestJava
> Task :processTestResources
> Task :testClasses
> Task :runDarttsLoginCheck
Gradle suite > Gradle test > runners.CucumberTestNGRunner > runScenario[0]("Login / Logoff Raven Dartts using Test Credentials", "Raven app Login") STANDARD_OUT
@DarttsLoginCheck
Scenario: Login / Logoff Raven Dartts using Test Credentials [90m# src/test/resources/features/common/Login.feature:11 [0m
23:33:01.840 [Test worker] INFO resources.base - driver setup is complete in before hook.
Embedding Screenshot [image/png 107848 bytes]
[32mGiven [0m [32m [0m [32m [1m"Admin" [0m [32m user login to app [0m [32m [1m"DARTTS" [0m [32m using [0m [32m [1m"appauth" [0m [90m# stepdefinitions.common.Common_Login.User_login_to_app(java.lang.String,java.lang.String,java.lang.String) [0m
[32mThen [0m [32mI should log out of Raven app [0m [90m# stepdefinitions.common.Common_Login.i_should_log_out_of_raven_app() [0m
23:33:16.419 [Test worker] INFO resources.base - driver teardown has completed.
Gradle suite > Gradle test > runners.CucumberTestNGRunner > runScenario[0]("Login / Logoff Raven Dartts using Test Credentials", "Raven app Login") PASSED
Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.0.
You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.
For more on this, please refer to https://docs.gradle.org/8.2/userguide/command_line_interface.html#sec:command_line_warnings in the Gradle documentation.
BUILD SUCCESSFUL in 22s
6 actionable tasks: 6 executed
[Pipeline] }
[Pipeline] // catchError
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Generate HTML report)
Stage "Generate HTML report" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Checking log)
Stage "Checking log" skipped due to earlier failure(s)
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
[GitHub Checks] GitHub check (name: Jenkins, status: COMPLETED) has been published.
ERROR: script returned exit code 1
Finished: FAILURE
If you notice after the first execution failed it ran the second time and passed the test. But still the build is marked as failed and also the email notification is not sent. Please help me in fixing this issue.
Jenkins does not allow to make the stage result better than it was recorded. So the fact that it runs the post block and skips the rest of the stages is expected and you can't do anything about it.
But you shouldn't, anyway.
Analyzing the build log is not an efficient way to solve this kind of problem. The error texts can change, and test logs are heavy - I'm not even sure that getLog() will manage to work with dozens or even hundreds of megabytes which is typical for tests to produce. Moreover, you can't even guarantee that the last 100 lines will contain the lines that you search for - any change of build logic can easily break this.
Restarting the tests without proper handling of statistics is obviously not a good practice because it will hide the flaky tests and pass the unstable application to UAT and production.
So, to make those flaky tests visible, you can do the following:
retry(count) in pipeline (either on a step or a stage level). That's the last resort option because it can be applied to a full step/stage only making the build long and restarting even the tests that don't need that, and the fact of retries will only be reflected in Blue Ocean UI.Putting the build commands into a try-catch-finally will lead to the same issues that were described above, so I don't put it to the list of solutions.
Also, there is no real need in highlighting the compilation errors because the test report will simply be empty in case of any.