Unable to run multiple command parellely in background on remote host using Jenkins

Question

I have a jenkins job DSL that run on a remote node (Linux OS) using "Restrict where this project can be run" Label.

It has "Build" step -> "Execute shell"

In the execute shell i have mentioned

sh /app/runalljobs.sh &

On the remote node host runalljobs.sh looks like below:

cat runalljobs.sh

ansible-playbook /app/test.yml -e argu=arg1
ansible-playbook /app/test.yml -e argu=arg2
.....
.....
ansible-playbook /app/test.yml -e argu=arg16

The runalljobs.sh is suppose to start 16 ansible processes in the background when runalljob.sh is executed.

This works fine when the script is executed manually from the remote node putty shell.

However, I want the script to start the ansible processes to run in the background on the remote node when invoked using jenkins job which is not happening.

I also tried commenting sh /app/runalljobs.sh &

and adding individual ansible command in the "Execute shell" as below:

ansible-playbook /app/test.yml -e argu=arg1 &
ansible-playbook /app/test.yml -e argu=arg2 &
.....
.....
ansible-playbook /app/test.yml -e argu=arg16 &

But this too did not trigger the ansible processes on the target node.

It works if i remove the "&" and then all the ansible commands runs serially one after the other on remote.

However, I wish all the ansible commands to be triggered parallely in the background and the Jenkins executor should proceed into executing other execute shell tasks.

Can you please suggest how can i acheive the requirement ?

Answer 1

Jenkins allows you to perform tasks in parallel, but there's a catch. This requires you switching to Jenkins Pipeline and then use of parallel. Then, your build script would look like:

pipeline {
    agent 'my-remote-machine'
    stages {
        ...
        stage('Ansible stuff') {
            parallel {
                stage('arg1') {
                    steps {
                        sh 'ansible-playbook /app/test.yml -e argu=arg1'
                    }
                }
                stage('arg2') {
                    steps {
                        sh 'ansible-playbook /app/test.yml -e argu=arg2'
                    }
                }
                ...
            }
        }
    }
}

If your command lines are quite similar (as in your example), you could use matrix section, to simplify the code:

matrix {
    axes {
        axis {
            name 'ARG'
            values 'arg1', 'arg2', 'arg3'
        }
    }
    stages {
        stage('test') {
            sh 'ansible-playbook /app/test.yml -e argu=${ARG}'
        }
    }
}

I know that this solution is a radical change - Jenkins Pipelines is a whole new world of CI. But it may be worth of effort because Pipelines are very promoted by Jenkins authors and lot of plugins are rewritten to work with them.