Search code examples
jenkinsgroovyparallel-processingjenkins-pipelinejenkins-declarative-pipeline

How to make parallel calls to the same function in stage in Jenkins Declarative Pipeline


What is the correct way to execute a function in a declarative pipeline in parallel?

Several posts on this forum are suggesting that you build an array, and pass the array to 'parallel' or pass a function to 'parallel'

Is it possible to create parallel Jenkins Declarative Pipeline stages in a loop?

Simple parallel execution in Jenkins for an array

I have made several attempts to get it to work but it always runs in serial

I think that the problem is that the function is being evaluated while I am building the array, even before I reach the 'parallel' step in the stage.

I wasn't able to arrive at a solution from the official documentation.

https://jenkins.io/blog/2017/09/25/declarative-1/

Here is how I would like it to work, but is it possible?

    pipeline {

        agent {
            label "l1" && "l2"
        }

        stages {

            stage ('Prepare Data') {
                // Some code that creates that data object
                // data is an array of maps
                data
            }

            stage ('Build') {
                script {
                    def stepsForParallel = [:]
                    data.each { d ->
                        // name is a key in the data map
                        stepsForParallel["${d['name']}"] = {
                            node {
                                stage("${d['name']}") {
                                    stepsForParallel[execDownStreamJob("DownStreamJobName", d)] = {
                                        println("Executing DownstreamJob")
                                    }
                                }
                            }
                        }
                    }
                    parallel stepsForParallel
                }
            }

        }
    }
}

The function below isn't really in the scope of this question, but I've added it in case it offers some value. It is explained best in this post: https://stackoverflow.com/a/42248825/5006720

// job is a string
// jobParameters is a hashmap
def execDownStreamJob(job, jobParameters) {

    // Some parsing takes place on hashMap and creates the p object 
    // 'p' is passed to the 'build job' function as a parameter

    def downStreamJobResult = build job: job, parameters: p, propagate: false

    // Some try/catch logic

}


Solution

  • the following part of code is really strange

    because you are trying to assign stepsForParallel twice

        stepsForParallel["${d['name']}"] = {
            node {
                stage("${d['name']}") {
                    stepsForParallel[execDownStreamJob("DownStreamJobName", d)] = {
                        println("Executing DownstreamJob")
                    }
                }
            }
        }
    

    i believe it should look like this (can't test it right now):

        script {
            def stepsForParallel = data.collectEntries{ d ->
                ["${d.name}", 
                    {
                        stage("${d.name}") {
                            println "DownstreamJob ${d.name} start")
                            execDownStreamJob("DownStreamJobName", d)
                            println "DownstreamJob ${d.name} end")
                        }
                    }
                ]
            }
            parallel stepsForParallel
        }