Gitlab CI pipeline file creates two pipelines at once

Question

I have the following .gitlab-cci.yml file configured:

image: node:latest

stages:
  - test
  - node
  - prepare
  - build
  - run

node_18:
  stage: node
  image: node:18.20.2
  script:
    - echo "Testing Node.js version 18.20.2"
    - npm i
    - cp config.example.json config.json
    - rm config.json

node_20:
  stage: node
  image: node:20.13.1
  script:
    - echo "Testing Node.js version 20.13.1"
    - npm i
    - cp config.example.json config.json
    - rm config.json

node_22:
  stage: node
  image: node:22.2.0
  script:
    - echo "Testing Node.js version 22.2.0"
    - npm i
    - cp config.example.json config.json
    - rm config.json

prepare:
  stage: prepare
  script:
    - echo "Removing node_modules..."
    - rm -rf node_modules/
  only:
    - master
    - merge_requests

compile:
  stage: build
  image: node:20
  script:
    - npm i
    - cp config.example.json config.json
    - node deploy-commands.js
    - npm run pm2start
    - npm run pm2stop
  only:
    - master
    - merge_requests

testrun:
  stage: run
  script:
    - npm run test
  only:
    - master
    - merge_requests

include:
  - template: Security/Secret-Detection.gitlab-ci.yml

Now, whenever I do a normal commit to any non-master branch, there is a pipeline created which runs two stages (test and node). However, when I make a PR/MR, it creates two pipelines, one being with those two stages that I mentioned, and the other with the other stages that are only flagged to run on merge requests and commits to master. Now, this is a problem because if the pipeline that started earlier fails, the MR will be able to be merged if the pipeline that started afterwards is all successful (which is a problem because I do my secret detection on all commits, so if there's a secret it will be overriden with the new pipeline that could be successfull). How can I make it so all of it is ran inside of one pipeline on mutual events (merge requests and commits to master).

If I make all of the stages run always, then they all run in one pipeline, but having them run like this separates them into two pipelines, which also slows everything down because I only have two runners.

Answer 1

I've updated your CI setup with a sample configuration you should be able to adapt to fit your use case. Remember that it's best to use rules instead of only/except as the latter is deprecated.

image: node:latest

stages:
  - test
  - node
  - prepare
  - build
  - run

node_18:
  stage: node
  image: node:18.20.2
  rules:
    - if $CI_PIPELINE_SOURCE == "merge_request_event"
  script:
    - echo "Testing Node.js version 18.20.2"
    - npm i
    - cp config.example.json config.json
    - rm config.json

node_20:
  stage: node
  image: node:20.13.1
  rules: !reference [.node_18, rules]
  script:
    - echo "Testing Node.js version 20.13.1"
    - npm i
    - cp config.example.json config.json
    - rm config.json

node_22:
  stage: node
  image: node:22.2.0
  rules: !reference [.node_18, rules]
  script:
    - echo "Testing Node.js version 22.2.0"
    - npm i
    - cp config.example.json config.json
    - rm config.json

prepare:
  stage: prepare
  rules: 
    - if: $CI_PIPELINE_SOURCE == "merge_request_event" || $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "master"
  script:
    - echo "Removing node_modules..."
    - rm -rf node_modules/

compile:
  stage: build
  image: node:20
  rules: !reference [prepare, rules]
  script:
    - npm i
    - cp config.example.json config.json
    - node deploy-commands.js
    - npm run pm2start
    - npm run pm2stop


testrun:
  stage: run
  rules: !reference [prepare, rules]
  script:
    - npm run test

include:
  - template: Security/Secret-Detection.gitlab-ci.yml

The reference keyword here is to reuse the config so you're not copy pasting the configuration across the file, similar to what YAML anchors do.

With that setup there will be two scenarios:

• Commit & branch is main: prepare -> build
• MR: node -> prepare -> build -> run