How to Automate Sync Between Azure DevOps and GitHub Repositories Using Azure Pipelines?

Question

Question:

I have a Azure DevOps Repository that is our primary source control and GitHub Repository that is being managed by an external team. Here’s the setup:

• Azure DevOps Repository: Contains Source/English.json and Source/German.json files.

• GitHub Repository: Contains WorkingFolder/English.json and WorkingFolder/German.json files.

These two files are almost the same in both repository except that the English.json file is maintained by me and the German.json file in the Github is updated by the external team based on the updates that I make in English.json file in the Github repository.

Current Workflow:

1. I manually Update English.json in Azure DevOps.
2. I manually apply the same updates to English.json in GitHub.
3. External team updates German.json with translations for the strings that I added in English.json file in GitHub.
4. I have to manually sync German.json from GitHub to Azure DevOps. I have to constantly check on for updates manually.

Goal: To Automate this process using Azure Pipelines and Github Action so that:

• Changes to English.json in Azure DevOps are automatically pushed to GitHub using Azure Pipelines.
• Changes to German.json in GitHub are automatically pulled into Azure DevOps using GitHub Action.

Solution that I am trying:

• Created an Azure YAML Pipeline that looks like below.

trigger:
  branches:
    include:
      - workingBranch
  paths:
    include:
      - Source/English.json

variables:
  branchName: 'workingBranch'

pool:
  vmImage: 'ubuntu-latest'

steps:
- checkout: self
  displayName: 'Checkout Azure DevOps Repository'
  ref: ${{ variables.branchName }}

- script: |
    git clone https://$(GITHUB_PAT)@github.com/your-org/GitTranslationRepo.git
    cd GitTranslationRepo
    git checkout development
    cp ../Source/English.json WorkingFolder/English.json
    git config user.name "your-username"
    git config user.email "your-email"
    git add WorkingFolder/English.json
    git commit -m "Add English.json from Azure DevOps for testing"
    git push origin development
  displayName: 'Sync English.json to GitHub development branch'
  env:
    GITHUB_PAT: $(GITHUB_PAT)

Issues Encountered:

• Authentication error: fatal: could not read Username for 'https://github.com': terminal prompts disabled.

Steps Taken:

• Created a GitHub PAT and stored it as a secret in Azure DevOps.
• Tried using the PAT directly in the script for authentication.

Questions:

1. How can I correctly configure the pipeline to authenticate with GitHub using a PAT or service connection?
2. Are there better approaches to automate this synchronization process between Azure DevOps and GitHub?

Any help or guidance would be greatly appreciated!

Answer 1

Repo structure

Sync from GitHub to Azure DevOps

1. Add a sync_to_azure.yml file to your GitHub development branch.

trigger:
  branches:
    include:
      - development
  paths:
    include:
      - WorkingFolder/German.json

pool:
  vmImage: ubuntu-latest

steps:
- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: |
      git clone https://$(System.AccessToken)@dev.azure.com/{ADO name}/{Project name}/_git/{Azure repo name}
      cd {Azure repo name}
      git checkout workingBranch
      cp -f ../WorkingFolder/German.json Source/German.json
      git config user.name "{user name}"
      git config user.email "{user email}"
      git add .
      git commit -m "Update German.json"
      git push origin workingBranch

  env:
    AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)

2. Create a pipeline in Azure DevOps.

   • Set your GitHub repo as resource and select the sync_to_azure.yml file in your development branch.

   

   • Change the default branch of your pipeline to development branch.

   

3. Change settings and permissions in Azure DevOps

   • Turn off “Protect access to repositories in YAML pipelines” in Organization Settings -> Settings.
   • Turn off “Protect access to repositories in YAML pipelines” in Project Settings -> Settings.
   • Set permissions for Build Service account in your Azure Repo. Go to Project settings -> Repositories -> Security -> Search your project Build Service account and set the permissions as shown in the screenshot. Its name format is " Build Service (Organization name)".

   

4. When you commit from WorkingFolder/German.json in your GitHub repo, the pipeline will be triggered to push the new German.json to your Azure repo.

   

Sync from Azure DevOps to GitHub

1. Add an azure_to_GitHub.yml file to your Azure repo workingBranch branch.

trigger:
  branches:
    include:
      - workingBranch
  paths:
    include:
      - Source/English.json

pool:
  vmImage: ubuntu-latest

steps:
- task: PowerShell@2
  inputs:
    targetType: 'inline'
    script: |
      git clone https://$(GitHubPAT)@github.com//{GitHub Name}/{GitHub Repo name}.git
      cd {GitHub Repo name}
      git checkout development
      cp -f ../Source/English.json WorkingFolder/English.json 
      git config user.name "{user name}"
      git config user.email "{user email}"
      git add .
      git commit -m "Update English.json"
      git push origin development

2. Create a pipeline in Azure DevOps using the above yaml file.

• Change the default branch of your pipeline to workingBranch branch.

• Add a secret variable to store your GitHub PAT.

  

3. When you commit from Source/English.json in your Azure repo, the pipeline will be triggered to push the new English.json to your GitHub repo.