Search code examples
terraformgitlab-cipipelinegitlab-ci-runnerscalr

How to make external stages in gitlab pipeline "wait" until all the gitlab internal stages are done?

I have defined 3 stages in gitlab-ci.yml. When there is a new commit, pipeline runs and these 3 stages run in parallel, which is expected and needed. (These stages run pre-requisite steps like security checks on code and other linting functions). I also have Scalr (another provider) inject external stages into the same pipeline (these run terraform policy checks and plan and apply).

However, the problem is that these external stages kick off in parallel of the internal stages mentioned above. I would like gitlab to pause any execution external stages until AFTER the internal (pre-req) stages have finished.

In case you are wondering, running the terraform plan and apply as gitlab internal stages is not an option.

Anyway to accomplish this?

Solution

  • GitLab has a super robust API. It's probably the way to go here.

    Terms

    I do want to standardize on some terms so I can make sure my recommendation is making sense. You say that your stages are running in parallel- but it is jobs that run in parallel within a given stage. For my response, I'm going to assume that you meant that you've a single stage on the 'internal repo' containing 3 jobs.

    Setup

    • Create a second stage in your 'internal repository' containing a single job.
    • This single job in the second stage will work as a synchronizer since the second stage will not begin until all jobs within the first stage complete.
    • This job should have a single activity, which is to call to your "external pipeline" using the GitLab jobs API. You'd configure a trigger to PLAY the job which was set to only be manual. https://docs.gitlab.com/ee/ci/triggers/
    • Configure the "external jobs" to be set to manual: true. This will prevent them from starting until they've been given the go-ahead

    Example

    stages:
  - test
  - remote_trigger

Linter:
  script:
    - echo "I linted lol!"
  stage: test

Security Check:
  script:
    - echo "I so secure!"
  stage: test

Start Terraform:
  script:
    - curl --request POST --form "token=$CI_JOB_TOKEN" --form ref=master "https://gitlab.example.com/api/v4/projects/9/trigger/pipeline"
  stage: remote_trigger

    This would create 3 jobs over 2 stages - Once all parallel jobs (security check & linter) in the first stage complete, the Terraform step can begin.