Syncing Github repository to AzureDevOps Repository
What Im trying to do is to sync repository in azuredevops to a repository in github. I did some googling and I stumbled into this article. It did not work right away but with minor modifications I was able to run the script from local Powershell successfully where the repository ADevops was populated from Github:
$AZUREDEVOPS_PAT = '*****'
$GITHUB_PAT = '****'
Write-Host ' - - - - - - - - - - - - - - - - - - - - - - - - -'
Write-Host ' reflect Azure Devops repo changes to GitHub repo '
Write-Host ' - - - - - - - - - - - - - - - - - - - - - - - - -'
$stageDir = "C:\temp"#'$(Build.SourcesDirectory)' | Split-Path
$githubDir = $stageDir +"\"+"gitHub"
$destination = $githubDir +"\"+"Temp_Folder"
#please provide your username
$alias = 'some-owner:'+ "${GITHUB_PAT}"
#Please make sure, you remove https from azure-repo-clone-url
$sourceURL = 'https://' + $alias + '@github.com/some-owner/Source-Repo'
$destURL = 'https://${AZUREDEVOPS_PAT}@dev.azure.com/Some_Org/Some_Porject/_git/Dest-Repo'
if((Test-Path -path $githubDir))
{
Remove-Item -Path $githubDir -Recurse -force
}
if(!(Test-Path -path $githubDir))
{
New-Item -ItemType directory -Path $githubDir
Set-Location $githubDir
Write-Output '*****Cloning Github remote secondary****'
git clone $sourceURL -q
}
else
{
Write-Host "The given folder path $githubDir already exists";
}
Set-Location $destination
Write-Output '*****Git removing remote secondary****'
#git remote rm -q secondary
Write-Output '*****Git remote add****'
git remote add --mirror=fetch secondary $destURL
Write-Output '*****Git fetch origin****'
git fetch $sourceURL -q
Write-Output '*****Git push secondary****'
git push secondary --all -q
Write-Output '**Azure Devops repo synced with Github repo**'
Set-Location $stageDir
if((Test-Path -path $githubDir))
{
Remove-Item -Path $githubDir -Recurse -force
}
After that I created an azure devops pipeline and stored the github and azure PAT into variables so the script become like this:
trigger:
- main
pool:
vmImage: ubuntu-latest
steps:
- script: echo Hello, world!
displayName: 'Run a one-line script'
- script: |
echo Add other tasks to build, test, and deploy your project.
echo See https://aka.ms/yaml
displayName: 'Run a multi-line script'
# Starter pipeline
- task: PowerShell@2
inputs:
targetType: 'inline'
script: |
Write-Host ' - - - - - - - - - - - - - - - - - - - - - - - - -'
Write-Host ' reflect Azure Devops repo changes to GitHub repo '
Write-Host ' - - - - - - - - - - - - - - - - - - - - - - - - -'
$stageDir = '$(Build.SourcesDirectory)' | Split-Path
$githubDir = $stageDir +"/"+"github"
Write-Host $githubDir
$destination = $githubDir +"/"+"Source-Repo"
#please provide your username
$alias = 'some-owener:'+ "$(GITHUB_PAT)"
#Please make sure, you remove https from azure-repo-clone-url
$sourceURL = 'https://' + $alias + '@github.com/some-owner/Source-Repo'
$destURL = 'https://$(AZUREDEVOPS_PAT)@dev.azure.com/Some_Org/Some_Project/_git/Dest-Repo'
if((Test-Path -path $githubDir))
{
Remove-Item -Path $githubDir -Recurse -force
}
if(!(Test-Path -path $githubDir))
{
New-Item -ItemType directory -Path $githubDir
Set-Location $githubDir
Write-Output '*****Cloning Github remote secondary****'
git clone $sourceURL -q
}
else
{
Write-Host "The given folder path $githubDir already exists";
}
Set-Location $destination
Write-Output '*****Git removing remote secondary****'
#git remote rm -q secondary
Write-Output '*****Git remote add****'
git remote add --mirror=fetch secondary $destURL
Write-Output '*****Git fetch origin****'
git fetch $sourceURL -q
#git pull secondary main
Write-Output '*****Git push secondary****'
git push secondary --all -q
Write-Output '**Azure Devops repo synced with Github repo**'
Set-Location $stageDir
if((Test-Path -path $githubDir))
{
Remove-Item -Path $githubDir -Recurse -force
}
Here is screenshot of how AZUREDEVOPS_PATH is defined:
When I run this script I get the following error on the git push towards the end:
fatal: could not read Password for 'https://***@dev.azure.com': terminal prompts disabled
When researching the error , a lot of post were suggesting to check "Allow scripts to access the Oauth token but I cant find it anywhere!!! Nor the running Agent. Is that something set on the pipeline itself or in azuredevops settings. Why Im getting this error if it works locally in powershell. Is there an easier alternative to what Im trying to do vs writing powershell script and creating pipeline?
Thanks
After struggling with this for sometime I was able to resolve with the following notes:
1- the issue turned out to be the PAT token itself not having the proper permission. Once I created new PAT and assign Full Access it worked! The remaining issue for me that I think Microsoft need to address is presenting more informative error message because none of the error messages I got indicated that its invalid permission wither in Azure devops pipeline or in powershell locally. All it was doing is trying to prompt me to re authenticate where in the pipeline its not able to do that because its disabled. In powershell if you cancel the authentication prompt this is the error message:
could not read Password for 'https://<PATP@dev.azure.com/ORG/PROJECT/_git/Repo': No such file or directory !!!
As you can see, there is no mentioning of permission issue.
2- In Regards to how to pass the PAT token , it seems there are couple of options out there and this since Microsoft has changed things few times in the last few years where its getting confusing.
I think before (and still working currently) you can do it as in:
https://<PATH>@dev.azure.com/Some_Org/Some_Porject/_git/Repo
However I always recommend you verify with Microsoft documentation because again things might change and by the time you read this who knows what formatting or authentication mechanism Microsoft promote. The current documentation recommends that you do it like this:
$MyPat = 'yourPat'
$headerValue = "Authorization: Basic " + [Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes(":" + $MyPat))
$env:GIT_AUTH_HEADER = $headerValue
git --config-env=http.extraheader=GIT_AUTH_HEADER clone https://dev.azure.com/yourOrgName/yourProjectName/_git/yourRepoNam
Now to add to the confusion, if you keep reading the referenced documentation under "Existing repos" you will find yet another format:
git remote add origin https://dev.azure.com/<PAT>@<company_machineName>:/<path-to-git-repo> path to git repo = <project name>/_git/<repo_name> git push -u origin --all
not sure what the <company_machineName> suppose to be, and what does it have to do with Git authentication. This format did not work to no matter how I try I keep getting error 400 invalid url.
3- However , if you are trying to do it from Azure Pipeline some post recommends to use System.AccessToken :
$env:GIT_AUTH_HEADER = "Authorization: Bearer $(System.AccessToken)"
but this one you need to make sure you set the right permission other wise you will get 403 Forbidden error see this post, Or copy whatever Id is given to you in the error message go to the repository security settings and in the search box paste the Id, whatever this Id is associated with youn need to set Allow for Contribute Permission and whatever permission you need. Goodluck finding any clear documentation from Microsoft about this.
4- There is 4th way you can authenticate but I dont recommend it because it has username & password in it and its not clear how long those credentials remain valid before they expire. Basically go the repository under Devops and click the Clone Button you will see Generate Git Credentials :
Click on the "" and you will be given username and password which you can use in the repo url in the format of :
https://<username>:<password>@dev.azure.com/ORG/PROJECT/_git/REPO
Im not sure what are the use cases for this but it works.Again Could not find any clear microsoft doc on this.
As you can see there are many of options , some are probably legacy and some are new. I think the pace in which things are changing from one version to another and lack of documentation makes things more confusing.
5- Finally, Why Im doing all this? It goes back to the simple scenario where I wanted to create a pipeline to sync Github repository to Azure Devops Repository.Unfortunately it doesnt seem that there is direct out of the box way to do it and you probably have to write the pipeline with all of the right Git commands. However how to trigger this pipeline so that it sync everytime you are making changes in github to certain branch there is very informative youtube about this that helped me a lot:
https://www.youtube.com/watch?v=YZlaoNPzaxA&t=206s
Thanks to Kevin Lu-MSFT for trying to help me with this.