Declarative Jenkins pipeline; running testcases in parallel batches

Question

At work we can only run unattended tests on a specially locked down jenkins instance, that I have no control over.

The job fetches the test collections from several repositories, so the amount of collections is very much dynamic, and only growing.

The test collections typically take 30 seconds to execute, so running them in parallel was nice, until the instance started running out of resources, and just began killing off processes.

The code below worked well, until we hit around 30 test collections

 script {
    echo " * *  find all postman collections, and create separate steps"
    def TMP_FILENAME = ".pm_files_list"
    sh "basename -s .postman_collection.json ./pm_collections/* > ${TMP_FILENAME}"
    def filenames = readFile(TMP_FILENAME).split( "\\r?\\n" );
    sh "rm -f ${TMP_FILENAME}"

    parallelStagesMap = filenames.collectEntries {
        ["${it}" : generateStage(it)]
    }
    parallel parallelStagesMap
}

where generateStage looks like this

def generateStage(collection) {
    return {
        stage("${collection}") {
            echo "running: ${collection}"
            sh  script: "npm run newman-run-collection --collection=\"${collection}\" "
        }
    }
}

I would like to achieve something like https://stackoverflow.com/a/67116586/10505992 so I came up with this: (My user does not have permissions to create jobs, so I couldn't use the other answer directly)

        stage('batch')
        {
            steps {
                script {
                    echo " * *  find all postman collections, and create seperate steps"
                    def TMP_FILENAME = ".pm_files_list"
                    sh "basename -s .postman_collection.json ./pm_collections/* > ${TMP_FILENAME}"
                    def collectionfiles = readFile(TMP_FILENAME).split( "\\r?\\n" ) as List;
                    //echo "collectionfiles: ${collectionfiles}"
                    collectionfiles.collate(10).each { List batch ->
                        echo "batch: ${batch}"

                        parallelStagesMap = batch.collectEntries {
                            ["${it}" : generateStage(it)]
                        }
                        parallel parallelStagesMap
                    }
                    parallelStagesMap["failFast"] = false
                }
            }
        }

But the issue is that only the first batch is executed. If I reduce the above to only echoing the lists, I see that collectionfiles and batch contain the filenames I expect.

I want to end up running the test collections in batches of 10 to 20 in each, but run the batches in parallel, just like the linked answer above.

Any help is much appreciated!

[Edit] Note for posterity; The reason why only first batch was executed, was that Jenkins did not proceed to next batch, if latest batch had a non-zero exit code. So grzegorzgrzegorz' elegant solution also only executed first batch, until I turned off non-zero exit codes in Newman. This means I'll have to think up another way of flagging failing test collections, since all steps will now look green in the Jenkins overview.

Answer 1

I had encountered the same problem at my work. Well, we always hit the resource limit. Anyway, you just need to wrap your parallel runner with sequential runner and feed it with proper data structure which should be list of lists:

["f1", "f2", "f3", "f4", "f5", "f6"] -> should be transformed to -> [["f1", "f2"], ["f3", "f4"], ["f5", "f6"]]

So, sequential runner is getting list of lists (LoL) and sequentially starts parallel runner for each item which is sublist. Thus, you can achieve parallel execution of specific number of items. Technically collate function comes in hand to make the transformation:

                list.collate(parallelSize)

So as the result you can adjust the amount of parallelism you can afford in your system:

pipeline
    {
        agent { label 'master' }
        parameters {
            string(name: 'Parallel_items', defaultValue: '2', description: '')
        }
        
        stages
        {
            stage('sandbox')
            {
                steps
                {
                    script
                    {
                        echo "start"
                        def fileNames = ["f1", "f2", "f3", "f4", "f5", "f6"]
                        runSequential(fileNames.collate(params.Parallel_items.toInteger()))
                    }
                }
            }
        }
    }
   
    
    def runSequential(parallelQueue){
        parallelQueue.each{
            runParallel(it)
        }
    }
    
    def runParallel(fileNames){
        parallelStagesMap = fileNames.collectEntries {
            ["${it}" : generateStage(it)]
        }
        parallel parallelStagesMap
    }
    
    def generateStage(collection) {
        return {
            stage("${collection}") {
                echo "running: ${collection}"
                sleep 5
            }
        }
    }