Best approach to run a number of asynchronous steps in sequence in a Jenkins pipeline

Question

Let me start with a brief description of the use case I want to implement.

Context

• I am developing a server application with Spring Boot and using maven to build it.
• The application needs a Redis database, so I can start the application once it has been built and the Redis DB is up and running.
• I am writing my REST API tests in JavaScript and I can run them once my app has been started.

Implementation

• For building the app, nothing special is required. I can simply do a mvn clean install. I can also do a mvn package to create an executable jar, which I can use in a Docker image.
• For running the app, I am using Docker Compose to express the dependency between my application and Redis. I can use docker-compose up to launch my environment. This is fine, but it is an asynchronous step: I don't know how much time it will take to have all services completely started an accepting requests.
• When I know that the application is ready, then I can launch my API tests written in JavaScript.
• At the end of the process, I can do a docker-compose down to tear down the test environment.

Question

• I am looking for advice on handling the asynchronous steps. In this simplified example, there is one step: how do I deal with the fact that it will take some time (say 5-15 seconds) for redis and the app to be fully started?
• Is there a feature of the Jenkins Pipeline plugin that I can use for this scenario? I was not able to find a lot of information in the tutorial or documentation. I have seen the waitUntil step, but I am looking for some guidance...
• Should I implement a simple shell script that issues an HTTP request to one of my app REST endpoints in a loop and that completes as soon as the app sends a response? Should I invoke this script in the waitUntil step?
• Is there a better approach to my requirement?

Answer 1

Should I implement a simple shell script that issues an HTTP request to one of my app REST endpoints in a loop and that completes as soon as the app sends a response? Should I invoke this script in the waitUntil step?

Yes, that is one legitimate approach.

You could also simply fold this into the script which runs your tests. To keep the Pipeline script short and to the point, and make it easier to test pieces of logic in isolation, store that script in SCM. Assuming it is in Bash (but Python or whatever would work fine):

while :
do
  if curl http://endpoint/
  then
    echo Up and running
    break
  else
    echo Still waiting
  fi
done
make test

and then your Pipeline script could read something like

node {
  stage 'build'
  checkout scm
  sh 'mvn clean install'
  stage 'test'
  sh 'docker-compose up'
  try {
    sh './run-tests-when-ready'
  finally {
    sh 'docker-compose down'
  }
}

Of course the call to docker-compose could even be put into such an external script if you like. Tricky to do the cleanup reliably in Bash (can use trap EXIT '…'), easier in a real language.