Unable to run az pipeline commands within Azure DevOps Task

Trying to dynamically retrieve all the variables from a variable group via Azure DevOps task in a YAML Pipeline. Originally tried leveraging the AzureCLI@2 task with the following code to retrieve the variableGroupID which would be used to get the variables inside of it:

$variableGroupId = $(az pipelines variable-group list --org $(System.CollectionUri) --project $(System.TeamProject) --query "[?name=='{{ parameters.variableGroupName }}'].id" -o tsv)

This command works locally but not when executing on a MS hosted agent like this:

parameters:
  variableGroupName: ''

steps:
- task: AzureCLI@2
  displayName: Azure CLI
  inputs:
    azureSubscription: ${{ parameters.azureSubscriptionName }}
    scriptType: pscore
    scriptLocation: inlineScript
    inlineScript: |
      az upgrade
      $variableGroupId = $(az pipelines variable-group list --org $(System.CollectionUri) --project $(System.TeamProject) --query "[?name=='{{ parameters.variableGroupName }}'].id" -o tsv)
      write-Host $variableGroupId
      $variables = $(az pipelines variable-group variable list --group-id $variableGroupId  --org $(System.CollectionUri)  --project $(System.TeamProject) -o yaml)
      write-Host $variables

This fails with the error:

Before you can run Azure DevOps commands, you need to run the login command (az login if using AAD/MSA identity else az devops login if using PAT token) to setup credentials. Please see https://aka.ms/azure-devops-cli-auth for more information

I have opened up an issue

In the meantime, I tried to run the commands to install the necessary pieces via scripts

   strategy:
      runOnce:
        deploy:
          steps:
          - task: AzureRmWebAppDeployment@3
            inputs:
              azureSubscription: Example - Dev
              WebAppName: wapp-Example-dev-eus
              Package: $(Pipeline.Workspace)/drop/Web.Example.zip
              TakeAppOfflineFlag: True
          - task: UsePythonVersion@0
            inputs:
              versionSpec: '3.x'
              architecture: 'x64'
          - task: CmdLine@2
            displayName: 'Upgrade pip'
            inputs:
              script: python -m pip install --upgrade pip
          - task: CmdLine@2
            displayName: 'upgrade azure cli'
            inputs:
              script: pip install --pre azure-cli --extra-index-url https://azurecliprod.blob.core.windows.net/edge
          - task: CmdLine@2
            displayName: 'Show Azure CLI version'
            inputs:
              script: az --version
          - task: CmdLine@2
            displayName: 'Install Azure DevOps Extension'
            inputs:
              script: az extension add -n azure-devops
          - task: CmdLine@2
            env:
              AZURE_DEVOPS_CLI_PAT: $(patCredential)
            displayName: 'Login Azure DevOps Extension'
            inputs:
              script: echo ${AZURE_DEVOPS_CLI_PAT} | az devops login
          - task: CmdLine@2
            displayName: 'Show List of Variables'
            inputs:
              script: |
                $variableGroupId = $(az pipelines variable-group list --org $(System.CollectionUri) --project $(System.TeamProject) --query "[?name=='{{ parameters.variableGroupName }}'].id" -o tsv)
                write-Host $variableGroupId
                $variables = $(az pipelines variable-group variable list --group-id $variableGroupId  --org $(System.CollectionUri)  --project $(System.TeamProject) -o yaml)
                write-Host $variables

However, when using both latest Ubuntu agents and those designated in the doc get an error:

WARNING: Failed to store PAT using keyring; falling back to file storage.
WARNING: You can clear the stored credential by running az devops logout.
WARNING: Refer https://aka.ms/azure-devops-cli-auth to know more on sign in with PAT.

I have opened up an issue with the documentation team as at the very least the provided steps do not work. Any assistance would be appreciated!

Solution

You can use the REST API instead of Azure CLI to get the information. It can be used with the standard tools already present on the Microsoft Hosted agents. It requires only vanilla powershell or powershell core, meaning to works on both windows and linux agents. The below example was successfully tested on windows-latest/windows-2019 and ubuntu-latest/ubuntu-20.04

The approach is the same as with Azure CLI.

List all available groups filtered by name to retrieve the variable group in question
Get all variables in the variable group using the variable group id from step

In fact, the pipeline also has an out of the box PAT token available with read access to variable groups. It is stored in the variable System.AccessToken. Using that instead of a manually managed one will further simplify things.

The script below is executed in a pwsh step, which is the built in Powershell task in Powershell core mode

- pwsh: |
    # Construct PAT authentication header
    $base64AuthInfo = [Convert]::ToBase64String([Text.Encoding]::ASCII.GetBytes(("{0}:{1}" -f "user",$env:SYSTEM_ACCESSTOKEN)))
    $headers = @{Authorization=("Basic {0}" -f $base64AuthInfo)}
    
    # Retrieve variable group id. Filter the result by setting the groupName query parameter
    $variableGroupId = $(Invoke-RestMethod -Headers $headers "$(System.CollectionUri)$(System.TeamProject)/_apis/distributedtask/variablegroups?groupName=${{ parameters.variableGroupName }}&api-version=6.0-preview.2").value[0].id
    
    # Retrieve variables in variable group with id $variableGroupId
    $variables = $(Invoke-RestMethod -Headers $headers "$(System.CollectionUri)$(System.TeamProject)/_apis/distributedtask/variablegroups/${variableGroupId}?api-version=6.0-preview.2").variables
    
    #Print variables as json (for demo purpose)
    $variables | ConvertTo-Json
  env:
    SYSTEM_ACCESSTOKEN: $(System.AccessToken)
  displayName: 'Retrieve variables'

Testing the above pipeline in a project on a variable group with two variables yields the following output:

{
  "Variable 1": {
    "value":  "Value 1"
  },
  "Variable 2": {
    "value":  "Value 2"
  }
}