How to combine multiple GitLab rules with And?

According to the documentation, multiple rules are being combined with OR operation, and indeed it behaves that way. But I have a specific situation.

So I have a set of jobs that are used for PCF platform and another set for Kubernetes platform. They are managed simply by setting variables. And it is possible that we may need both to be active:

.pcf-platform:
  rules:
    - if: $PLATFORM_PCF == "true"

.k8s-platform:
  rules:
    - if: $PLATFORM_K8S == "true"

.manual-feature-branch-optional:
  rules:
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "schedule" && $CI_PIPELINE_SOURCE != "merge_request_event"
      when: manual
      allow_failure: true

.manual-default-branch-optional:
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "schedule"
      when: manual
      allow_failure: true

Now we imagine we have these jobs:


deploy-dev-pcf:
  extends:
    - .manual-feature-branch-optional
    - .pcf-platform
  script:
    - do deploy

deploy-dev-k8s:
  extends:
    - .manual-feature-branch-optional
    - .k8s-platform
  script:
    - do deploy

deploy-prod-pcf:
  extends:
    - .manual-default-branch-optional
    - .pcf-platform
  script:
    - do deploy

Now, imagine both PLATFORM_K8S and PLATFORM_PCF are set to true and the current branch is a feature branch (not default); as expected, both deploy-dev-pcf and deploy-dev-k8s are added to the pipeline but so as deploy-prod-pcf.

It seems the first rule that checks if the current branch is default branch is resolved to false but then it evaluates the next rule that checks PLATFORM_PCF equals to true and since it is true it adds the job while we are still in a feature branch.

I need to somehow make these two separate rules being And-ed so I can turn on/off jobs depending on the desired platform as well as the type of branch.

I could create one set of rules for Kubernetes and one set for PCF but that increases the number of rules unreasonably and I am reluctant to that approach.

In the documentation it is saying that only clauses are being evaluated with And but it seems only is being deprecated in favour of rules.

How can I achieve this?

Solution

In order to get this right, you'll need to understand:

How jobs are merged together; and
How rules are evaluated

Or you can just skip to the last heading for your answer :-)

How jobs are merged together

Part of your problem here seems to be your expectation of what extends: does in this case. When array values meet through extends:, they override one another and do not combine. See merge details for additional context.

For example, given this configuration:

.foo:
  rules:
    - if: $FOO == "foo"

.bar:
  rules:
    - if: $BAR == "bar"
job:
  extends:
    - foo
    - bar  # array of rules here will override the previous declarations

In other words, only one set of those rules will be effective in the resulting configuration in this case, as if written like so:

job:
  rules:
    - if: $BAR == "bar"

So, in your example, only the platform rule is being applied, which results in the behavior you are observing.

One way to get around this is to use !reference instead to build your rules array:

rules:
  - !reference [.foo, rules]
  - !reference [.bar, rules]

How rules are evaluated

Another aspect that needs understanding here is how rules are evaluated. GitLab will evaluate each rule in order and stop at the first rule that evaluates true. There can be at most, one rule that takes effect! If no rules match, the job is excluded from the pipeline.

So, there are a couple ways you can get the effect of AND evaluation.

You can combine it into a single rule and combine with && operators:

rules:
  - if: $FOO == "foo" && $BAR == "bar"

You can also get the effect you want by ordering different rules in a particular order and using the inverse logic combined with when: never. For example, these rules have the same effect as above:

rules:
  - if: $FOO != "foo"
    when: never
  - if: $BAR != "bar"
    when: never
  - when: on_success  # base case is important!

Inverting the logic here (checking != instead of ==), we allow evaluation to continue to subsequent rules. This is important if you want to define and re-use rules separately.

In practice

Applying the understandings explained above, using your example, you might do something like this which allows you to define each rule just once:


# define different sets of rules
.rules:
  pcf-only:
    - if: $PCF_PLATFORM != "true"
      when: never
  k8s-only:
    - if: $K8S_PLATFORM != "true"
      when: never
  manual-feature-branch-optional:
    - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH && $CI_PIPELINE_SOURCE != "schedule" && $CI_PIPELINE_SOURCE != "merge_request_event"
      when: manual
      allow_failure: true

# Combine these rules in different ways that can be reused
.k8s-deploy:
  rules: # Order matters!
    - !reference [.rules, k8s-only]
    - !reference [.rules, manual-feature-branch-optional]

.pcf-deploy:
  rules:
    - !reference [.rules, pcf-only]
    - !reference [.rules, manual-feature-branch-optional]

# Use `extends:` for each combination of rules as-needed
deploy-dev-pcf:
  extends:
    - .pcf-deploy  
  script:
    - do deploy

# alternatively, you can define rules directly instead of using `extends:`
deploy-dev-k8s:
  rules:
    - !reference [.rules, k8s-only]
    - !reference [.rules, manual-feature-branch-optional]
  script:
    - do deploy

# ... and so on

There's probably other refactoring opportunities for you here, but that's perhaps beside the point of the question :)