ADO yaml pipeline - Iterating over a splitted string fails

Question

I am scratching my head about the following pipeline and can't really figure out what the issue is.

I have the following yaml pipeline (code stripped to the issue):

stages:
- stage: Check

  variables:
    - name: trackBranch
      value: 'theBranch'

  jobs:
  - job: Checking
    displayName: Check branch

    steps:
    - powershell: |
        git fetch --depth=2 origin +refs/heads/${{ variables.trackBranch }}:refs/remotes/origin/${{ variables.trackBranch }}
        $diffFolders=git diff --name-only HEAD^ HEAD | ForEach-Object { $_.Trim() } | ForEach-Object { [System.IO.Path]::GetDirectoryName($_) } | Sort-Object -Unique
        
        Write-Output "Updated folders: $diffFolders"

        $namesArray = @()
        foreach ($entry in $diffFolders) {
          $namesArray += $entry
        }
        $names = $namesArray -join ','
        $namesAsString = $names | Out-String
        Write-Output "##vso[task.setvariable variable=changedFolders;isOutput=true]$namesAsString"
      displayName: 'searching'
      name: compare

  - job: Iteration
    dependsOn: Checking
    variables:
      - name: updatedMods
        value: $[dependencies.Checking.outputs['compare.changedFolders']]
      - name: testMods
        value: "mod1,mod2,mod3"
    steps:
      - script: |
          echo $(updatedMods)

      - ${{ each entry in split(variables.updatedMods, ',') }}:
        - script: echo ${{ entry }}
          displayName: "Processing ${{ entry }}"
      
      - ${{ each entry in split(variables.testMods, ',') }}:
        - script: echo ${{ entry }}
          displayName: "Processing ${{ entry }}"

In the first job it is comparing two commits from a branch, to find out if something got updated. For example it would return an object with the content: folder2 folder5 folder8, because these folders got changed. Then I am "forcing" it into a string write it to the variable 'changedFolders'.

In the next job, the variable gets echoed and I can see the content of the variable as expected. But the iteration part is failing with:

dependencies.Checking.outputs['compare.changedFolders']: Syntax error: Invalid arithmetic operator. (Error causing character is ".Checking.outputs['compare.changedFolders']")

But the iteration and splitting with the testMods is working fine.

So either I don't see where the issue is or the "received" variable in the second job is not really a string and is causing the trouble.

I searched for similar issues here on SO, but did not find a similar issue.

Answer 1

This is because the each loop ${{ each entry in split(variables.updatedMods, ',') }} is a template expression, which is processed at compile time. Anything that happens after the pipeline starts running is after template compilation. So, the value of variable updatedMods is defined at the compile time with this string $[dependencies.Checking.outputs['gitCompare.changedFolders']]. It will not changed to the output value from job 1.

For more information, you can refer to Runtime expression syntax.

As a workaround, you can iterate over and split the string $(updatedMods) in a task.

Sample task:

      - script: |
          echo $(updatedMods)

          # Use a for loop to iterate over each item in the string
          for folder in $(updatedMods)
          do
            echo $folder
          done

Test result:

Update:

I have to figure out something else, because in each iteration some executions of further tasks should happen. (not only echoing stuff).

I can share another workaround. In this workaround, the two jobs in your current pipeline will run in separate pipelines.

In pipeline one, your job: Checking runs and you will get the diff folders. Then we can use the Runs - Run Pipeline REST API in the job to run the pipeline two and pass the diff folders value to the pipeline two. In pipeline two, you can set a Runtime parameter. The parameter value is passed from the pipeline one.

If that's acceptable to you, the following are the specific steps.

1. Pipeline one yaml:

trigger:
- none
stages:
- stage: Check

  variables:
    - name: trackBranch
      value: 'main'
  jobs:
  - job: Checking
    displayName: Check branch
    steps:
    - powershell: |
        # your steps to get the diff folders here
        $diffFolders="folder11,folder52,folder82" 
       
        $organization = "your organization name"
        $project="your projectname"
        $pipelineId=49    # you can find the pipeline definition Id of pipeline two from the pipeline URL.        

        $body = @"
        {
            "templateParameters": {
                "Mods": "$diffFolders"
            }
        }
        "@
        $response = Invoke-RestMethod "https://dev.azure.com/$organization/$project/_apis/pipelines/$($pipelineId)/runs?api-version=6.1-preview.1" -Method 'POST' -Headers @{Authorization = "Bearer $(System.AccessToken)"} -Body $Body -ContentType application/json
        $response | ConvertTo-Json

2. Pipeline two yaml. You can add your further tasks in pipeline two.

trigger:
- none

pool:
  vmImage: ubuntu-latest

parameters:
- name: Mods
  type: string
  default: []
steps:
- ${{ each entry in split(parameters.Mods, ',') }}:
  - script: echo ${{ entry }}
    displayName: "Processing ${{ entry }}"

3. Note: You should add your build service account the permission to Edit queue build configuration and Queue builds of pipeline two. Otherwise, you may see errors like "message": "TF215106: Access denied. projectname Build Service (orgname) needs Queue builds permissions for build pipeline 49 in team project to perform the action.

4. Test result: Pipeline two is triggered by pipeline one and get the Mods value from pipeline one.