Loop through secrets in a devops pipeline

Question

I'm building a pipeline to deploy a web app and migrate some tenant databases.

I've got variables defined in my library which pull secrets in from keyvault. If I enable diagnostic logging on the pipeline I can see output like

##[debug]Promise for downloading secret value for: Test1  
Downloading secret value for: Test1.  
##[debug]Promise for downloading secret value for: Test2  
Downloading secret value for: Test2.  
##[debug]Promise for downloading secret value for: DATABASE-URL-FOO-TEST  
Downloading secret value for: DATABASE-URL-FOO-TEST.  
##[debug]Promise for downloading secret value for: DATABASE-URL-FOO-TEST-ALT  
Downloading secret value for: DATABASE-URL-FOO-TEST-ALT. 
...
##[debug]set DATABASE-URL-FOO-TEST-ALT=********
##[debug]Processed: ##vso[task.setvariable variable=DATABASE-URL-FOO-TEST-ALT;isOutput=false;issecret=true;]***   
##[debug]set DATABASE-URL-FOO-TEST-ALT=********  
##[debug]Processed: ##vso[task.setvariable variable=DATABASE-URL-FOO-TEST-ALT;isOutput=false;issecret=true;]***  
##[debug]set Test2=********  
##[debug]Processed: ##vso[task.setvariable variable=Test2;isOutput=false;issecret=true;]***  
##[debug]set Test2=********  

Now what I'd really like to do is for in over all the variables starting DATABASE-URL- and pass that URL to my migration script. I've tried a few things but I can't seem to find my variables in the environment. This bash task gives me no output from the for loops.

 - task: Bash@3
  inputs:
    targetType: 'inline'
    script: |
      echo "all vars"
      for var in "${!variable@}"; do
          echo "$var=${!var}"
      done

      for key in "${!VAR[@]}"; do
        echo "$key"
        if [[ $key == SECRET_DATABASE-URL* ]]; then
         
         DB_URL=${!VAR[$key]} echo "$DB_URL"
         DB_URL=${!VAR[$key]} ./bin/migrate-databases.sh
        fi
      done

In the diagnostic output from that task I can see lines like

##[debug]loading SECRET_DATABASE-URL-FOO-TEST-ALT
##[debug]loading SECRET_SYSTEM_ACCESSTOKEN
##[debug]loading SECRET_TEST2
##[debug]loading SECRET_DATABASE-URL-FOO-TEST
##[debug]loading SECRET_TEST1

so it looks like they're being set. How can I get to these variables?

My pipeline is running on vmImage:ubuntu-latest hence the use of Bash, but I'm open to any other method of achieving this as long as it's compatible with the Ubuntu agent.

Answer 1

Now what I'd really like to do is for in over all the variables starting DATABASE-URL- and pass that URL to my migration script. I've tried a few things but I can't seem to find my variables in the environment.

Unlike a normal variable, secret variables are not automatically decrypted into environment variables for scripts. You cannot directly get them from environment(doc here).

Checked on my side, it's able to get the secret variables name with Azure CLI, task below:

- bash: |
    # Get all variables from the variable group
    variables=$(az pipelines variable-group variable list --group-id 11 --output json)

    # Get the "DATABASE-URL-" started variables name
    vars=$(echo "$variables" | jq -r 'to_entries[] | select(.key | startswith("DATABASE-URL-")) | .key')
    echo "$vars"

    for var in $vars
    do
      echo "checking: $var"              # output the variable name
      #echo $($var)                       # it doesn't work
      #echo "${!var}"                     # it doesn't work
    done
  displayName: 'Get Secret Variables values'
  env:
    AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

However, with the variable name in script, you cannot get the variable value. For example,$($var) and ${!var} doesn't work, even remove hyphen - in the variable name to simplify the variable name.

I'm afraid it's not supported to dynamically get the expected secret variable values in DevOps.