Not able to commit changed files on windows-latest agent in Azure Devops Pipeline to repository

Question

To clarify, i want to push the changes made by my PowerShell scripts in the Azure Pipeline back to the same branch and repository from where the code was cloned. I use a personal access token url to push to the repository.

The PowerShell scripts will edit JSON files and replace values in the JSON files, this is working correctly so files are definitely changed.

This is currently my code in the YAML pipeline. I replaced critical values with placeholder x and X.

trigger: none

pool:
  vmImage: 'windows-latest'

variables:
  # Path vars
  PoliciesMainBicepFilePath: "$(Agent.BuildDirectory)/s/TestBicepPoliciesEnv/main.bicep"
  PoliciesMainParametersFilePath: "$(Agent.BuildDirectory)/s/TestBicepPoliciesEnv/main.parameters.jsonc"
  ResourceGroup: "RG_XXXXX_IaC"
  AzureResourceManagerConnection: 'XXXXXXX'

  # GIT stuff
  SubscriptionId: "9536d2f5-xxxx-4cf8-ac41-xxxxxxxxxx"
  jsonFilePath: '$(System.DefaultWorkingDirectory)\TestBicepPoliciesEnv\customPolicies\'
  repoUrl: 'https://[email protected]/XXXXX%20INFRA%20IaC/_git/XXXXX%20INFRA%20IaC'
  branchName: '$(Build.SourceBranch)'
  commitMessage: "Updating_parameters_in_Policy_JSON_files"
  #SD
  System.Debug: true

parameters:

- name: Tag1Name
  displayName: 'Required Tag 1 name'
  type: string
  default: 'Owner'
- name: Tag2Name
  displayName: 'Required Tag 2 name'
  type: string
  default: 'Environment'
- name: Tag3Name
  displayName: 'Required Tag 3 name'
  type: string
  default: 'App'
- name: Tag4Name
  displayName: 'Required Tag 4 name'
  type: string
  default: 'Region'

stages:
  - stage: "Deploy_policies"
    jobs:
      - job: Build_policies
        displayName: Customization of the policies
        steps:
          - checkout: self
          
          - powershell: |
              Write-Host "executing replaceTagsPolicy.ps1"
              ls
              .\TestBicepPoliciesEnv\replaceTagsPolicy.ps1 -Tag1Name $env:Tag1Name -Tag2Name $env:Tag2Name -Tag3Name $env:Tag3Name -Tag4Name $env:Tag4Name
            env:
              Tag1Name: ${{ parameters.Tag1Name }}
              Tag2Name: ${{ parameters.Tag2Name }}
              Tag3Name: ${{ parameters.Tag3Name }}
              Tag4Name: ${{ parameters.Tag4Name }}

      - job: Push_to_repository
        dependsOn: Build_policies 
        displayName: Push JSON file changes to the repository
        steps:
          - powershell: |
              git checkout ${{ variables.branchName }}
              git config --global user.email "[email protected]"
              git add ${{ variables.jsonFilePath }}
    
              # Commit changes
              git commit -m ${{ variables.commitMessage }}

              # Display status
              git status

              # Display remote
              git remote -v

              # Display branch
              git branch

              # Display logs
              git log -n 5

              # Configure Git to use PAT for authentication
              git remote set-url origin ${{ variables.repoUrl }}

              
              # Push changes to the same branch
              git push origin ${{ variables.branchName }}

These are the logs by git:

error: pathspec 'refs/heads/TestPolicies' did not match any file(s) known to git
HEAD detached at origin/xxxxxxxx9acaa213c8e7d86385f05fd43721
nothing to commit, working tree clean
HEAD detached at origin/xxxxxxxx9acaa213c8e7d86385f05fd43721
nothing to commit, working tree clean
origin  https://xxxxxxx-be.visualstudio.com/XXXXX%20INFRA%20IaC/_git/XXXXX%20INFRA%20IaC (fetch)
origin  https://xxxxxxx-be.visualstudio.com/XXXXX%20INFRA%20IaC/_git/XXXXX%20INFRA%20IaC (push)
* (HEAD detached at origin/xxxxxxxx9acaa213c8e7d86385f05fd43721)
commit xxxxxxxx9acaa213c8e7d86385f05fd43721
Author: xxxxx.xxxxxxxx <[email protected]>
Date:   Tue Nov 14 14:55:47 2023 +0000

    Update pipelineDeployPolicies.yml for Azure Pipelines
error: src refspec refs/heads/TestPolicies does not match any
error: failed to push some refs to 'https://xxxxxx-be.visualstudio.com/XXXXX%20INFRA%20IaC/_git/XXXXX%20INFRA%20IaC'

Anyone knows how i can change my script so it pushes the changed JSON files to the same repository? PAT has full access so thats not the issue.

Answer 1

Based on your requirement, please test the steps below:

1. The 2 PowerShell tasks to replace tags and push commits should be within 1 single agent job; when initializing each agent job, you will be assigned a reimaged agent or a different agent; hence, the downstream agent job will checkout your repo again by default without acknowledging the updates in the upstream job;
2. Use the predefined variable $(Build.SourceBranchName) instead of $(Build.SourceBranch) to define ${{ variables.branchName }}; their values are different;
3. Use git checkout -b ${{ variables.branchName }}, which according to this document

If -B is given, <new-branch> is create if it doesn’t exist; otherwise, it is reset. This is the transactional equivalent of

$ git branch -f branch [start-point]

$ git checkout branch

4. Here is the sample YAML for your reference.

variables:
  jsonFilePath: '$(System.DefaultWorkingDirectory)'
  repoUrl: 'https://$(System.AccessToken)@dev.azure.com/$(AzureDevOpsOrgName)/$(TheProjectName)/_git/$(TheRepoName)'
  branchName: '$(Build.SourceBranchName)'
  commitMessage: "Updating_parameters_in_Policy_JSON_files-$(Build.BuildId)"
  #SD
  System.Debug: true

stages:
- stage: Deploy_policies
  jobs:
  - job: Build_policies
    displayName: Customization of the policies
    steps:
      - checkout: self
      - powershell: |
          $json = Get-Content -Path "definition.json" -Raw | ConvertFrom-Json
          $json.message = "Commit mesessage by $(Build.BuildId)"
          $json | ConvertTo-Json -Depth 100 | Set-Content -Path "definition.json"
        name: ModifyJson
        
      - powershell: |
          git checkout -b ${{ variables.branchName }}
          git config --global user.email "[email protected]"
          git add ${{ variables.jsonFilePath }}
          # Commit changes
          git commit -m ${{ variables.commitMessage }}
          # Configure Git to use PAT for authentication
          git remote set-url origin ${{ variables.repoUrl }}
          
          # Push changes to the same branch
          git push origin ${{ variables.branchName }}

5. In the sample, you can use PAT or $(System.AccessToken) to authenticate. I tested and succeeded to push the commits during agent job with both methods. If you would like to use $(System.AccessToken) which is the token generated by the pipeline service account, please check the job authentication scope and make sure the corresponding service account has sufficient permission to push to your repo/branch.

Hope the information may help.