Gitlab CI CD only run Pipeline Step when certain criteria is met

I am trying to configure my CI/CD pipeline so that a certain step is only executed if some conditions are met.

My current rule definition looks like this:

 rules:
    - if: '($CI_PIPELINE_SOURCE == "push" || $CI_PIPELINE_SOURCE == "web" || $CI_PIPELINE_SOURCE == "merge_request_event") && $CI_COMMIT_BRANCH == "develop" '
      exists:
        - $MY_FILE

What I want to achieve:

Execute step only if a merge from branch_x was done into develop
Or Execute step when pipeline triggered over the GitLab UI
Or Execute step if a commit is pushed directly to develop
And the file in $MY_FILE is present in the repository

Other than expected, the step gets not executed on

Push
Merge Branch_X into develop
Manually trigger over GitLab UI

The corresponding file exists within the repository.

The step work is if I change back to

  only:
    - develop

But then I can not set the condition that the file needs to be present.

Solution

Part of the issue you're running into is that some Predefined Variables only exist in certain types of pipelines. For example, the CI_COMMIT_TAG variable, which would hold the name of a tag, only exists if the pipeline is a "Tag pipeline" (ran after a tag is created). However, if a tag points to a commit that is the HEAD of a branch that is the Source of a Merge Request, it is still just a Tag pipeline, so none of the Merge Request specific variables will exist.

Due to this, your conditional will get slightly more complex since we'll need some more parenthesis, or more if clauses for our rules section:

rules:
  - if: ($CI_PIPELINE_SOURCE == 'push' || $CI_PIPELINE_SOURCE == 'web') && $CI_COMMIT_REF_NAME == 'develop'
    exists:
      - $MY_FILE
    when: always
  - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'develop'
    exists:
      - $MY_FILE
    when: always
  - when: never

Note: you could write this using only one if clause and just use more parenthesis.

Let's look at these changes one by one.

First, the push and web pipeline sources are more or less identical in terms of how the pipeline is ran, and which Predefined Variables are available.

Second, the variable $CI_COMMIT_BRANCH is only available in certain pipelines. Most noticeably, it only exists when there is a Branch. If the pipeline runs for a tag, the variable will not exist, it's not just empty. If it's a merge request event, the variable will not exist. $CI_COMMIT_REF_NAME is more reliable, even though it can have more values (Commit SHAs, Branch names, or Tag names).

Third, the merge_request_event source is a totally different animal compared to the other sources we're dealing with here. The Predefined Variables available are totally different since there are now two branches (the source and the target). Also, a merge_request_event pipeline can only ever run if you have rules in your pipeline definition for merge_request_event's.

I ran some tests with this rules scenario, with this pipeline definition:

stages:
  - run

Run Job:
  stage: run
  image: alpine:latest
  script:
    - echo $CI_PIPELINE_SOURCE
    - echo $CI_COMMIT_REF_NAME
    - echo $CI_COMMIT_BRANCH
    - echo $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
    - echo $CI_MERGE_REQUEST_TARGET_BRANCH_NAME

Second Run Job:
  stage: run
  image: alpine:latest
  rules:
    - if: ($CI_PIPELINE_SOURCE == 'push' || $CI_PIPELINE_SOURCE == 'web') && $CI_COMMIT_REF_NAME == 'develop'
      exists:
        - 'a_file'
      when: always
    - if: $CI_PIPELINE_SOURCE == 'merge_request_event' && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == 'new_branch'
      exists:
        - 'a_file'
      when: always
    - when: never  
  script:
    - echo $CI_PIPELINE_SOURCE
    - echo $CI_COMMIT_REF_NAME
    - echo $CI_COMMIT_BRANCH
    - echo $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
    - echo $CI_MERGE_REQUEST_TARGET_BRANCH_NAME

Here's the job output for a normal Push event to a branch other than develop:

Executing "step_script" stage of the job script
00:02
$ echo $CI_PIPELINE_SOURCE
push
$ echo $CI_COMMIT_REF_NAME
other_branch
$ echo $CI_COMMIT_BRANCH
other_branch
$ echo $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
$ echo $CI_MERGE_REQUEST_TARGET_BRANCH_NAME

Note: only the first job runs since the branch isn't develop. If the branch is develop and a_file exists, both jobs run, and the output is identical.

Here's the output of a job with the 'web' source, to a branch other than develop:

Executing "step_script" stage of the job script
00:00
$ echo $CI_PIPELINE_SOURCE
web
$ echo $CI_COMMIT_REF_NAME
other_branch
$ echo $CI_COMMIT_BRANCH
other_branch
$ echo $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
$ echo $CI_MERGE_REQUEST_TARGET_BRANCH_NAME

Again, in this case there is only one job in the pipeline. If instead we hit Run Pipeline for the develop branch and a_file exists, we run both jobs and the output is identical:

Executing "step_script" stage of the job script
00:00
$ echo $CI_PIPELINE_SOURCE
web
$ echo $CI_COMMIT_REF_NAME
develop
$ echo $CI_COMMIT_BRANCH
develop
$ echo $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
$ echo $CI_MERGE_REQUEST_TARGET_BRANCH_NAME

Lastly, if we push to a branch that's the source of a merge request targeting develop with a_file in the repository, we get the merge_request_event, but only get the second job due to the way Merge Request Pipelines work:

Executing "step_script" stage of the job script
00:01
$ echo $CI_PIPELINE_SOURCE
merge_request_event
$ echo $CI_COMMIT_REF_NAME
some_other_branch
$ echo $CI_COMMIT_BRANCH
$ echo $CI_MERGE_REQUEST_SOURCE_BRANCH_NAME
some_other_branch
$ echo $CI_MERGE_REQUEST_TARGET_BRANCH_NAME
new_branch

Here's the documentation for Merge Request Pipelines for more information.