How to build one specific Docker image in a mono repo that contains many Dockerfiles

I have a repository with 8 different Dockerfiles that builds different images. They are all in the same repo because they share 80% of the same code. Each Dockerfile corresponds to a separate product. I currently have a azure-pipeline.yaml file to build the 8 Dockerfiles into images. When making a PR or merging into my main branch, I typically only updated a feature for one of my 8 products. So it makes no sense to update all 8 images.

I am wondering what are my options for triggering just the build of the product I changed. But occasionally, trigger all 8 to be build (if a shared method is updated). For example, is there a flag feature I can use to say when this flag is identified, run this part of the pipeline only?

Side note: I know I can split my mono-repo into many different repos, and create a package that allows sharing of the common code. However, this will require a large time commitment, wondering if there is an easier way.

Solution

You can use a combination of path filters and conditionals in your yaml file to selectively run builds based on changes in the pull request or commit.

1. Path Filters

You can use path filters to specify which paths should trigger the pipeline. For example, if each Dockerfile is in a separate directory, you could have something like this:

trigger:
  paths:
    include:
      - 'productA/*'

This would trigger the build only when there are changes in the productA directory.

2. Conditional Steps

You can add conditions to your steps or jobs to decide whether they should run or not. For example, you can define variables and use them to conditionally run steps:

variables:
  isFullBuild: $[eq(variables['Build.Reason'], 'Manual')]

steps:
- script: echo Building ProductA...
  condition: or(eq(variables.isFullBuild, true), changesContain('productA/*'))

You can combine the 2 to add a triggers and condition your pipeline like below.

trigger:
  branches:
    include:
    - main
  paths:
    include:
    - 'productA/*'
    - 'productB/*'
    - 'common/*'

- job: BuildProductA
  condition: or(eq(variables.isFullBuild, true), changesContain('productA/*', 'common/*'))
  steps:
  - script: echo Building ProductA...

- job: BuildProductB
  condition: or(eq(variables.isFullBuild, true), changesContain('productB/*', 'common/*'))
  steps:
  - script: echo Building ProductB...

More information: changeContains is not an Expression in Azure Pipelines (It's only a placeholder to add a similar condition). Before using the Job Condition, You should add a Job which runs a script and generates an Output which will hold changed folders. Which should be used to compare if the changes exist in the condition for job. Example below:

CheckChanges.ps1

$folders = @("productA", "productB", "productC") # specify your folder names
$changedFolders = @()

foreach ($folder in $folders) {
    $changes = git diff --name-only HEAD HEAD~1 -- $folder
    if ($changes) {
        $changedFolders += $folder
    }
}

# Join the changed folders into a comma-separated string
$changedFoldersString = $changedFolders -join ','

# Set the pipeline variable
Write-Output "##vso[task.setvariable variable=changedFolders;isOutput=true]$changedFoldersString"

Your updated pipeline may look like:

trigger:
  branches:
    include:
    - main
  paths:
    include:
    - 'productA/*'
    - 'productB/*'
    - 'common/*'


- job: CheckChanges
  pool:
    vmImage: 'windows-latest'
  steps:
  - powershell: |
      .\CheckChanges.ps1
    name: checkChangesStep

- job: BuildProductA
  dependsOn: CheckChanges
  variables:
    changes: $[ dependencies.CheckChanges.outputs['checkChangesStep.changedFolders'] ]
 
  condition: or(eq(variables.isFullBuild, true), contains(variables['changes'], 'productA'))
  steps:
  - script: echo Building ProductA...

- job: BuildProductB
  dependsOn: CheckChanges
  variables:
    changes: $[ dependencies.CheckChanges.outputs['checkChangesStep.changedFolders'] ]
 
  condition: or(eq(variables.isFullBuild, true), contains(variables['changes'], 'productB'))
  steps:
  - script: echo Building ProductB...

Note: I have not validated the above pipeline by running it and may contain some syntax/other issues, and is provided only as a general approach.