Multiple execution of newman in a single stage on Jenkins

Question

I have some collections made with Postman and want to execute them using Newman in the same stage on a Jenkins environment.

My Jennkinsfile looks like below:

sh 'newman run /xxx/collection1.json'
sh 'newman run /xxx/collection2.json'
sh 'newman run /xxx/collection3.json'

However, if any one of a test in the first collection fails, the shell exits with 1 so neither collection 2 nor 3 will be run.

I tried adding || true at the end of each shell execution, but this makes the resulting exit code always be 0, so I can't see the error from my dashboard even if some of the tests fail.

sh 'newman run /xxx/collection1.json || true'
sh 'newman run /xxx/collection2.json || true'
sh 'newman run /xxx/collection3.json || true'

Is there any way to know the final test result while letting all the shells run?

Answer 1

The only approach that comes to my mind would be to wrap the testing stage in certain sub-stages that runs within a parallel block. The following approach should work:

pipeline {
  agent any
    options {
      buildDiscarder logRotator(artifactDaysToKeepStr: '', artifactNumToKeepStr: '', daysToKeepStr: '', numToKeepStr: '5')
    }
  stages {
    stage('First') {
      steps {
        sh 'echo "This stage is going to run very first"'
      }
    }
    stage('Hello') {
      failFast false
      parallel {
        stage('one') {
          steps {
            sh 'echo one; exit 1'
          }
        }
        stage('two') {
          steps {
            sh 'echo two'
          }         
        }
        stage('three') {
          steps {
            sh 'echo three'
          }         
        }
      }
    }
  }
}

Stage "First" is supposed to run first, the 3 other stages will run in parallel, with stage "two" and "three" running successfully although stage "one" failed. The whole build is marked as failed, but stage "two" and "three" are marked as green in the dashboard.

EDIT: If you're not interested in displaying failures of single stages you can wrap your shell steps in a try/catch block, write the result as a Boolean in global variables and check them afterwards. Like this:

Boolean run1 = true
Boolean run2 = true
Boolean run3 = true

pipeline {
  agent any
  stages {
    stage('Hello') {
      steps {
        script {
          try {
            sh "echo one; exit 1"
          } catch (any) {
            run1 = false
          }
          try {
            sh "echo two"
          } catch (any) {
            run2 = false
          }
          try {
            sh "echo three"
          } catch (any) {
            run3 = false
          }
          if (run1 == false || run2 == false || run3 == false) {
            currentBuild.result = 'FAILURE'
          }
        }
      }
    }
  }
}

But in your case the most elegant way would be to wrap every newman run in a catchError block:

pipeline {
  agent any
  stages {
    stage('First') {
      steps {
        catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
          sh "echo one; exit 1"
        }
      }
    }
    stage('Second') {
      steps {
        catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
          sh "echo two"
        }
      }
    }
    stage('Third') {
      steps {
        catchError(buildResult: 'SUCCESS', stageResult: 'FAILURE') {
          sh "echo three"
        }
      }
    }
  }
}