Search code examples
jenkinsjenkins-declarative-pipeline

Reuse parallel stage in declarative Jenkinsfile

I have a declarative Jenkins pipeline which contains several identical stages running in parallel. I would like to reuse these sections, however so far the best I've been able to achieve is as follows:

stage('Deploy & E2E') {
    parallel {
        stage('E2E 1') {
            agent { ... }
            environment { ... }
            steps {
                script { runE2ESteps() }
            }
            post { ... }
        }
        stage('E2E 2') {
            agent { ... }
            environment { ... }
            steps {
                script { runE2ESteps() }
            }
            post { ... }
        }
        stage('Other unrelated stage, not wanting reuse') {
            agent { ... }
            environment { ... }
            steps {
                // something else
            }
            post { ... }
        }
    }
}

I am able to partially reuse bits, by extracting the steps out into a method, but there is still quite a bit of duplication in the agent, environment and post blocks. Ideally I want to write something like:

stage('Deploy & E2E') {
    parallel {
        script {
            for (int i = 0; i < 5; i++) {
                stage('E2E ${i}') {
                    agent { ... }
                    environment { ... }
                    steps { ... }
                    post { ... }
                }
            }
        }
        stage('Other unrelated stage, not wanting reuse') {
            agent { ... }
            environment { ... }
            steps {
                // something else
            }
            post { ... }
        }
    }
}

But apparently I can't do this in declarative land - Jenkins seems very finicky about what precisely you're allowed to have at various points in the file.

I've tried a few other approaches, too, but none of them have worked:

  • Using matrix for the E2E block seemed promising, but you're not allowed to nest matrix within parallel so this didn't pan out (because there are other, unrelated stages we want running in parallel to the E2E steps)
  • Using the in-line parallel command. This doesn't achieve what I want because I need each stage running in its own agent (not parallelised within the same agent).

I've attached a diagram of what the pipeline looks like - it's behaving exactly as I want right now, just with more code duplication than I'd like. If we want to change the parallelism of our E2E, it currently involves copy+pasting or removing a block of code, rather than (ideally) changing one number somewhere.

Maybe this just isn't possible, but I figured I might as well chuck out a post to see if there are any ideas that I missed.

Diagram

Solution

  • We never found a solution to this in Jenkins. In the end, we've moved to Github Actions where this is possible with a matrix job.