So I am trying to set up my YAML file to run a build on my 2 Databases only if there are changes in their respective git folder. Which the bash script below correctly runs and provides out the True/False flags.
Bash Script:
set -x
pathFilterDB1="DB1/"
pathFilterDB2="DB2/"
changedFiles=$(git diff --name-only HEAD~1..HEAD)
DB1Changed=false
DB2Changed=false
if [[ -n "$changedFiles" ]]; then
for file in $changedFiles; do
[[ "$file" == $pathFilterDB1* ]] && DB1Changed=true
[[ "$file" == $pathFilterDB2* ]] && DB2Changed=true
done
fi
echo "DB1_CHANGED: $DB1Changed"
echo "STAGING_CHANGED: $DB2Changed"
echo "##vso[task.setvariable variable=DB1_CHANGED;isOutput=true]$DB1Changed "
echo "##vso[task.setvariable variable=DB2_CHANGED;isOutput=true]$DB2Changed "
And the Azure Pipeline condition below seems to pull output, but is always pulling an additional 2 Single " ' "
condition: eq(dependencies.CheckChanges.outputs[bashchanges.DB1_CHANGED], 'true')
condition: eq(dependencies.CheckChanges.outputs[bashchanges.DB2_CHANGED], 'true')
Here is what it evaluates as:
Evaluating: eq(dependencies['CheckChanges']['outputs']['bashchanges.DB1_CHANGED'], 'true')
Expanded: eq('true''', 'true')
Result: False
Evaluating: eq(dependencies['CheckChanges']['outputs']['bashchanges.DB2_CHANGED'], 'true')
Expanded: eq('true''', 'true')
Result: False
This condition is done for 2 different jobs.
When I change the DB1 Condition below:
condition: eq(dependencies.CheckChanges.outputs[bashchanges.DB1_CHANGED], 'true''')
condition: eq(dependencies.CheckChanges.outputs[bashchanges.DB1_CHANGED], 'true')
I get these responses:
Evaluating: eq(dependencies['CheckChanges']['outputs']['bashchanges.DB1_CHANGED'], 'true''')
Expanded: eq('true', 'true''')
Evaluating: eq(dependencies['CheckChanges']['outputs']['bashchanges.DB2_CHANGED'], 'true')
Expanded: eq('false', 'true')
I for the life of me can't figure out if it's an additional ' somewhere or if there's something else I should be looking for.
trigger:
branches:
include:
- dev
- test
- prod
pool:
vmImage: 'windows-2022'
variables:
buildConfiguration: 'Release'
dacpacLocation: 'D:\a\1\s\Staging\bin\Release\Staging.dacpac'
storageAccountName: 'azseastorageasrdev'
containerName: 'asrbackup'
blobName: 'staging_$(Build.BuildId)_$(Date:yyyyMMddHHmmss).dacpac'
storageAccountKey: 'key'
${{ if eq(variables['Build.SourceBranchName'], 'prod') }}:
serverName: 'database.windows.net'
stagingDatabaseName: 'azsea-rda-prod-staging-db'
reportingDatabaseName: 'azsea-rda-prod-reporting-db'
${{ if eq(variables['Build.SourceBranchName'], 'test') }}:
serverName: '.database.windows.net'
stagingDatabaseName: 'azsea-rda-test-staging-db'
reportingDatabaseName: 'azsea-rda-test-reporting-db'
${{ if eq(variables['Build.SourceBranchName'], 'dev') }}:
serverName: 'azsea-rda-dev-sql.database.windows.net'
stagingDatabaseName: 'azsea-rda-dev-staging-db'
reportingDatabaseName: 'azsea-rda-dev-reporting-db'
jobs:
- job: CheckChanges
displayName: 'Check for changes'
steps:
# Checkout code and fetch the full Git history
- checkout: self
fetchDepth: 0 # Fetch all commit history, not just the latest commit
- task: Bash@3
inputs:
targetType: 'inline'
script: |
set -x
pathFilterReporting="Reporting/"
pathFilterStaging="Staging/"
changedFiles=$(git diff --name-only HEAD~1..HEAD)
reportingChanged=false
stagingChanged=false
if [[ -n "$changedFiles" ]]; then
for file in $changedFiles; do
[[ "$file" == $pathFilterReporting* ]] && reportingChanged=true
[[ "$file" == $pathFilterStaging* ]] && stagingChanged=true
done
fi
echo "REPORTING_CHANGED: $reportingChanged"
echo "STAGING_CHANGED: $stagingChanged"
echo "##vso[task.setvariable variable=REPORTING_CHANGED;isOutput=true]$reportingChanged "
echo "##vso[task.setvariable variable=STAGING_CHANGED;isOutput=true]$stagingChanged "
name: bashchanges
- job: BuildAndDeployReporting
displayName: "Build and Deploy Reporting DB"
dependsOn: CheckChanges
condition: eq(dependencies.CheckChanges.outputs[bashchanges.REPORTING_CHANGED], 'true')
steps:
- script: echo "Building and Deploying Reporting DB"
- task: PowerShell@2
inputs:
targetType: 'inline'
scriptType: 'Inline'
inlineScript: |
$netFramework472Url = "https://download.visualstudio.microsoft.com/download/pr/13815836/10ad8c401f59cda96ab9a5c155cdc394/ndp472-devpack-enu.exe"
$netFramework472Installed = Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full' -ErrorAction SilentlyContinue
if ($null -eq $netFramework472Installed -or $netFramework472Installed.Release -lt 461808) {
Write-Host ".NET Framework 4.7.2 is not installed. Installing..."
Invoke-WebRequest -Uri $netFramework472Url -OutFile "$env:TEMP\ndp472-devpack-enu.exe"
Start-Process -FilePath "$env:TEMP\ndp472-devpack-enu.exe" -ArgumentList "/quiet /norestart" -Wait
Write-Host ".NET Framework 4.7.2 installation complete."
}
else {
Write-Host ".NET Framework 4.7.2 is already installed."
}
displayName: 'Install .NET Framework 4.7.2'
- task: NuGetCommand@2
inputs:
command: 'restore'
restoreSolution: 'Reporting/Reporting.sqlproj'
displayName: 'Restore NuGet packages'
- task: VSBuild@1
inputs:
solution: 'Reporting/Reporting.sqlproj'
msbuildArgs: '/p:DeployOnBuild=true /p:PublishProfile=$(buildConfiguration)'
platform: 'Any CPU'
configuration: '$(buildConfiguration)'
- task: PublishPipelineArtifact@1
inputs:
targetPath: '$(Build.ArtifactStagingDirectory)'
artifactName: 'DatabaseBuildArtifacts'
publishLocation: 'pipeline'
- publish: D:\a\1\s\Reporting\bin\Release\
artifact: dacpac
displayName: 'Publish DACPAC Artifact'
- script: |
echo 'Uploading DACPAC to Azure Blob Storage...'
az storage blob upload --account-name "$(storageAccountName)" --container-name "$(containerName)" --name "$(blobName)" --file "$(dacpacLocation)" --account-key "$(storageAccountKey)" --overwrite
displayName: 'Upload DACPAC to Azure Storage'
- task: SqlAzureDacpacDeployment@1
inputs:
azureSubscription: 'Azure Resource - Main - Subscription Level'
authenticationType: 'server'
serverName: '$(serverName)'
databaseName: '$(reportingDatabaseName)'
sqlUsername: 'username'
sqlPassword: 'pwd'
deployType: 'DacpacTask'
dacpacFile: '$(dacpacLocation)'
additionalArguments: '/p:BlockOnPossibleDataLoss=true'
IpDetectionMethod: 'AutoDetect'
DeleteFirewallRule: true
- job: BuildAndDeployStaging
displayName: 'Build and Deploy Staging DB'
dependsOn: CheckChanges
condition: eq(dependencies.CheckChanges.outputs['bashchanges.STAGING_CHANGED'], 'true')
steps:
- script: echo "Building and Deploying Staging DB"
- task: PowerShell@2
inputs:
targetType: 'inline'
scriptType: 'Inline'
inlineScript: |
$netFramework472Url = "https://download.visualstudio.microsoft.com/download/pr/13815836/10ad8c401f59cda96ab9a5c155cdc394/ndp472-devpack-enu.exe"
$netFramework472Installed = Get-ItemProperty -Path 'HKLM:\SOFTWARE\Microsoft\NET Framework Setup\NDP\v4\Full' -ErrorAction SilentlyContinue
if ($null -eq $netFramework472Installed -or $netFramework472Installed.Release -lt 461808) {
Write-Host ".NET Framework 4.7.2 is not installed. Installing..."
Invoke-WebRequest -Uri $netFramework472Url -OutFile "$env:TEMP\ndp472-devpack-enu.exe"
Start-Process -FilePath "$env:TEMP\ndp472-devpack-enu.exe" -ArgumentList "/quiet /norestart" -Wait
Write-Host ".NET Framework 4.7.2 installation complete."
}
else {
Write-Host ".NET Framework 4.7.2 is already installed."
}
displayName: 'Install .NET Framework 4.7.2'
- task: NuGetCommand@2
inputs:
command: 'restore'
restoreSolution: 'Staging/Staging.sqlproj'
displayName: 'Restore NuGet packages'
- task: VSBuild@1
inputs:
solution: 'Staging/Staging.sqlproj'
msbuildArgs: '/p:DeployOnBuild=true /p:PublishProfile=$(buildConfiguration)'
platform: 'Any CPU'
configuration: '$(buildConfiguration)'
- task: PublishPipelineArtifact@1
inputs:
targetPath: '$(Build.ArtifactStagingDirectory)'
artifactName: 'DatabaseBuildArtifacts'
publishLocation: 'pipeline'
- publish: D:\a\1\s\Staging\bin\Release\
artifact: dacpac
displayName: 'Publish DACPAC Artifact'
- script: |
echo 'Uploading DACPAC to Azure Blob Storage...'
az storage blob upload --account-name "$(storageAccountName)" --container-name "$(containerName)" --name "$(blobName)" --file "$(dacpacLocation)" --account-key "$(storageAccountKey)" --overwrite
displayName: 'Upload DACPAC to Azure Storage'
- task: SqlAzureDacpacDeployment@1
inputs:
azureSubscription: 'Azure Resource - Main - Subscription Level'
authenticationType: 'server'
serverName: '$(serverName)'
databaseName: '$(stagingDatabaseName)'
sqlUsername: 'username'
sqlPassword: 'pwd'
deployType: 'DacpacTask'
dacpacFile: '$(dacpacLocation)'
additionalArguments: '/p:BlockOnPossibleDataLoss=true'
IpDetectionMethod: 'AutoDetect'
DeleteFirewallRule: true
I can reproduce the same issue when using the same YAML code as yours.
The reason of the issue that you give an extra whitespace to the values of the output variables when using the setvariable
command to set the output variables.
echo "##vso[task.setvariable variable=REPORTING_CHANGED;isOutput=true]$reportingChanged "
echo "##vso[task.setvariable variable=STAGING_CHANGED;isOutput=true]$stagingChanged "
After removing the extra whitespace, the extra '
get removed from the values of the output variables, and the condition
can get the correct the output values.
echo "##vso[task.setvariable variable=REPORTING_CHANGED;isOutput=true]$reportingChanged"
echo "##vso[task.setvariable variable=STAGING_CHANGED;isOutput=true]$stagingChanged"
EDIT:
In the Bash script, changing the command lines
from
[[ "$file" == $pathFilterReporting* ]] && reportingChanged=true
[[ "$file" == $pathFilterStaging* ]] && stagingChanged=true
to
if [[ "$file" == $pathFilterReporting* ]]; then
reportingChanged=true
fi
if [[ "$file" == $pathFilterStaging* ]]; then
stagingChanged=true
fi
does not make any effect.
I attempted the following Bash script that directly set the values on the setvariable
command, but it still will have an extra '
character on the values sometimes.
- bash: |
set -x
pathFilterReporting="Reporting/"
pathFilterStaging="Staging/"
changedFiles=$(git diff --name-only HEAD~1..HEAD)
if [[ -n "$changedFiles" ]]; then
for file in $changedFiles; do
if [[ "$file" = $pathFilterReporting* ]]; then
echo '##vso[task.setvariable variable=REPORTING_CHANGED;isoutput=true]true'
fi
if [[ "$file" = $pathFilterStaging* ]]; then
echo '##vso[task.setvariable variable=STAGING_CHANGED;isoutput=true]true'
fi
done
fi
name: bashchanges
Based on my further investigation, the problem seems is with Bash (or Bash task) itself. When I convert the Bash script to be a PowerShell script like as below, and then execute the PowerShell script using a PowerShell task (pwsh
or powershell
), everything can always work fine.
- pwsh: |
$pathFilterReporting = "Reporting/"
$pathFilterStaging = "Staging/"
$changedFiles = git diff --name-only HEAD~1..HEAD
foreach ($file in $changedFiles)
{
if ($file -match "^$pathFilterReporting.*")
{
Write-Host "##vso[task.setvariable variable=REPORTING_CHANGED;isoutput=true]true"
}
if ($file -match "^$pathFilterStaging.*")
{
Write-Host "##vso[task.setvariable variable=STAGING_CHANGED;isoutput=true]true"
}
}
name: bashchanges
With above configuration:
When there are modified files contained in the Reporting folder, set the output variable REPORTING_CHANGED
with true
value. The condition
on the BuildAndDeployReporting
returns True
, and the job gets run.
Expanded: eq('true', 'true')
Result: True
When there are not modified files contained in the Reporting folder, not set the output variable REPORTING_CHANGED
. The condition
on the BuildAndDeployReporting
job returns False
, and the job gets skipped.
Expanded: eq(Null, 'true')
Result: False
When there are modified files contained in the Staging folder, set the output variable STAGING_CHANGED
with true
value. The condition
on the BuildAndDeployStaging
returns True
, and the job gets run.
Expanded: eq('true', 'true')
Result: True
When there are not modified files contained in the Staging folder, not set the output variable STAGING_CHANGED
. The condition
on the BuildAndDeployStaging
job returns False
, and the job gets skipped.
Expanded: eq(Null, 'true')
Result: False