Azure DevOps YAML deployment pipeline cannot reference variable group in stage

Question

I'm trying to set up a build and deployment pipeline for Azure Data Factory content. I have the build part working but I want to use a secondary YAML template for the deployment to different environments. When I create individual variables for each environment it works as expected but when I try to use a variable group my variables are not recognized. Below is the code that doesn't work:

trigger:
  branches:
    include:
      - main
  paths:
    exclude:
      - CICD/*

pool:
  vmImage: ubuntu-latest

variables:
  - group: VG-DWH-GEN
  - name: BuildAdfResourceId
    value: /subscriptions/$(subscriptionID)/resourceGroups/$(ResourceGroupNamePrefix)$(environmentID)/providers/Microsoft.DataFactory/factories/$(ADFNamePrefix)$(environmentID)-01
  - name: WorkspaceArmTemplateDirectory
    value: $(Pipeline.Workspace)/adf-artifact-ArmTemplate

stages:
- stage: Deploy_Dev_Stage
  displayName: Deploy Dev stage
  dependsOn: Build_ADF_ARM_Stage
  condition: succeeded('Build_ADF_ARM_Stage')

  variables:
  - group: VG-DWH-DEV

  jobs:
  - template: ADF_Deployment.yml
    parameters: 
      azureSubscription: $(ServiceConnectionPrefix)$(variables.environmentID)-development
      environment: $(variables.environment)
      resourceGroupName: '$(ResourceGroupNamePrefix)$(variables.environmentID)'
      dataFactoryName: '$(ADFNamePrefix)$(environmentID)-01'

I get an error message like this:

Declaring the variable explicitly instead of the group does work. I've tried changing the syntax from $() to ${{}} and $[] but none work. It seems that it just does not recognize the variables within the group. I'm at a loss. All the documentation I am finding states I should be able to use variable groups at stage level. The pipeline has been authorized to use the variable group so that's not the issue either. Any help would be greatly appreciated.

NB: I know the stage " Build_ADF_Arm_Stage" is being references. I've let that part of the YAML out of the question as that part is working as expected.

Update: As requested the content of the variable group:

The variable group is located in the same project as the pipeline.

The content of the template yaml:

parameters: 
  - name: azureSubscription
    type: string
  - name: environment
    type: string
  - name: resourceGroupName
    type: string
  - name: dataFactoryName
    type: string

jobs:
  - deployment: 'deployment_job_${{ parameters.environment }}'
    displayName: 'Deployment Job ${{ parameters.environment }}'
    environment: 'Deploy ADF to ${{ parameters.environment }}'
    variables:
      - group: 'VG-DWH-${{ upper(parameters.environment) }}'
    strategy:
      runOnce:
        deploy:
          steps:
            - task: DownloadPipelineArtifact@2
              displayName: Download Build Artifacts - ADF ARM Templates
              inputs:
                artifactName: 'adf-artifact-ArmTemplate'
                targetPath: $(WorkspaceArmTemplateDirectory)
            - script: dir
              displayName: List files in Workspace
              workingDirectory: '$(WorkspaceArmTemplateDirectory)'

            - task: AzurePowerShell@5 
              displayName: 'Azure PowerShell script: Pre-deployment' 
              inputs: 
                azureSubscription: '${{ parameters.azureSubscription }}'
                ScriptPath: '$(WorkspaceArmTemplateDirectory)/PrePostDeploymentScript.ps1'
                ScriptArguments: '-armTemplate ''$(WorkspaceArmTemplateDirectory)/ARMTemplateForFactory.json'' -ResourceGroupName ${{ parameters.resourceGroupName }} -DataFactoryName ${{ parameters.dataFactoryName }} -predeployment $true -deleteDeployment $false' 
                azurePowerShellVersion: LatestVersion

            - task: AzureResourceManagerTemplateDeployment@3 
              displayName: 'ARM Template deployment: Data Factory' 
              inputs:
                azureResourceManagerConnection: '${{ parameters.azureSubscription }}'
                subscriptionId: '$(subscriptionID)'
                resourceGroupName: '${{ parameters.resourceGroupName }}' 
                location: 'West Europe' 
                csmFile: '$(WorkspaceArmTemplateDirectory)/ARMTemplateForFactory.json' 
                csmParametersFile: '$(WorkspaceArmTemplateDirectory)/ARMTemplateParametersForFactory.json' 
                overrideParameters: >
                  -factoryName "${{ parameters.dataFactoryName }}"
                  -LS_ADLS_Dataplatform_properties_typeProperties_url "https://$(StorageAccount).dfs.core.windows.net/"
                  -LS_ASQLDB_Dataplatform_properties_typeProperties_server "$(ServerName)"
                  -LS_REST_AFAS_properties_typeProperties_url "$(AFASRestAPIURL)"
                  -LS_KV_Dataplatform_properties_typeProperties_baseUrl "$(keyVaultURL)"
                deploymentMode: 'Incremental'
 
            - task: AzurePowerShell@5 
              displayName: 'Azure PowerShell script: Post-deployment' 
              inputs: 
                azureSubscription: '${{ parameters.azureSubscription }}'
                ScriptPath: '$(WorkspaceArmTemplateDirectory)/PrePostDeploymentScript.ps1'
                ScriptArguments: '-armTemplate ''$(WorkspaceArmTemplateDirectory)/ARMTemplateForFactory.json'' -ResourceGroupName ${{ parameters.resourceGroupName }} -DataFactoryName ${{ parameters.dataFactoryName }} -predeployment $false -deleteDeployment $true' 
                azurePowerShellVersion: LatestVersion

Answer 1

Setting the service connection

Variables in a variable group are referenced at runtime using macro syntax or a runtime expression.

But service connections, on the other hand, are resolved at compile time so the following won't work if it references a runtime variable (e.g. from a variable group):

jobs:
  - template: ADF_Deployment.yml
    parameters: 
      azureSubscription: $(ServiceConnectionPrefix)$(variables.environmentID)-development
      # ...

You must use an hard-coded value or a variable that can be resolved at compile time:

variables:
  myServiceConnection: foobar

jobs:
  - template: ADF_Deployment.yml
    parameters: 
      azureSubscription: ${{ variables.myServiceConnection }}
      # ...

Using variable macro syntax

To reference a variable using macro syntax use $(myvar).

So instead of:

$(variables.environmentID)

Try:

$(environmentID)

See Understand variable syntax for more details.