Dynamically generate variable group names in Azure Pipelines

Question

I want to dynamically generate a variable group name on Azure pipelines, based on a variable that is coming from a variable group. I found several questions related to this, but none seems to address this specific use case.

So far, this has been working for us with the following code:

variables:
  - group: release

jobs:
- job: blue
  condition: eq(variables['deployment'], 'blue')
  variables:
  - group: blue-vars
  steps:
    - template: ../templates/some-code.yml

- job: green
  condition: eq(variables['deployment'], 'green')
  variables:
  - group: green-vars
  steps:
    - template: ../templates/some-code.yml

But now we need to extend this implementation, and we will have 10-12 jobs from whose only one will execute, according to the $(deployment) variable (coming from the release variable group). That is ugly, and difficult to maintain if we will need 10-12 blocks doing the exact same, with the same variables having different values, so I would like to refactor this solution.

I tried the obvious solution (below), but it doesn't work most likely because the $(deployment) variable is only available on runtime, and needs to be available on compilation time.

- job: a_single_colour_1
  variables:
  - group: $(deployment)-vars
  steps:
    - template: ../templates/some-code.yml

I tried with compile-time expressions, but it seems that variables coming from variable groups are not resolved before compiling the expressions, so the below attempt is also failing:

- job: a_single_colour_2
  variables:
  - ${{ if eq(variables['deployment'], 'blue') }}:
    - group: blue-vars
  - ${{ if eq(variables['deployment'], 'green') }}:
    - group: green-vars
  steps:
    - template: ../templates/some-code.yml

After going through the documentation, I decided to use runtime expressions - as shown below - but it fails due to an error in the YAML (seems not to be supported):

 job: a_single_colour_3
  variables:
  - $[ if eq(variables['deployment'], 'blue') ]:
    - group: blue-vars
  - $[ if eq(variables['deployment'], 'green') ]:
    - group: green-vars
  steps:
    - template: ../templates/some-code.yml

Lastly, I tried to use a job template (code below), and I can see clearly that the variable $(deployment) is also not being resolved, causing a compile-time error due to the incompatible value for the deployment parameter.

On the main pipeline

- template: job_template.yml
  parameters: 
    deployment: $(deployment)

job_template.yml

parameters:
- name: deployment
  type: string
  values:
  - green
  - blue

jobs:
- job: a_single_colour_4
  variables:
  - name: deployment
    value: ${{ parameters.deployment }}
  - ${{ if eq(variables['deployment'], 'blue') }}:
    - group: blue-vars
  - ${{ if eq(variables['deployment'], 'green') }}:
    - group: green-vars
  steps:
    - template: ../templates/some-code.yml

Any idea how can I achieve what I want?

Answer 1

I want to dynamically generate a variable group name on Azure pipelines, based on a variable that is coming from a variable group

My first suggestion would be to redesign your pipeline to avoid this situation, e.g. maybe using a pipeline parameter instead of the first variable group (release)?

In case that's not possible, and in order to reduce code duplication, you can use a loop to generate all possible jobs and use conditions to define which one will execute, according to the value of $(deployment).

Example:

trigger: none

pool:
  vmImage: 'ubuntu-latest'

parameters:
  - name: colors
    displayName: 'All possible values for the colors'
    type: object
    default:
      - green
      - blue

variables:
  - group: release

jobs:
  - ${{ each color in parameters.colors }}:
    - job: ${{ color }}
      condition: eq(variables['deployment'], '${{ color }}')
      variables:
      - group: ${{ color }}-vars
      steps:
        - checkout: none
        - script: echo "Hello, $(color)!"
          displayName: 'Display color from variable group'

Running the job when $(deployment) is set to green: