When Azure Pipeline pulls code from Bitbucket cloud, why is the local working directory inconsistent?

Question

I have a Azure DevOps (ADO) pipeline with a runtime variable or parameter called option. option can have values of build, test, ci, or cd. The pipeline immediately calls a template based on the value of option. Each of these templates then immediately calls another template, setup_tasks.yml. The setup_tasks.yml template immediately checks out the source code from Bitbucket cloud.

There's only one setup_tasks.yml. It's used by all four templates.

Contents of `azure-pipelines.yml':

# Azure Pipeline for ACM builds.

trigger:
- azure-pipelines
- parallel-builds2

parameters:
- name: option
  type: string
  default: 'release-build'
  values:
  - 'ci'
  - 'cd'
  - 'release-test'
  - 'release-build'
- name: agent
  type: string
  default: 'ACMBUILD3'
  values:
  - 'ACMBUILD3'
  - 'ACMBUILD4'
  - 'ACMBUILD5'
  - 'ACMBUILD6'
  - 'ACMBUILD32'
  - 'ACMBUILD64'
- name: bits
  type: string
  default: '64'
  values:
  - '64'
  - '32'

pool:
  name: 'Default'

variables:

  - name: buildConfiguration
    ${{ if eq(parameters.option, 'cd')}}:
      value: 'CD'
    ${{ if eq(parameters.option, 'ci')}}:
      value: 'Debug'
    ${{ if eq(parameters.option, 'release-build')}}:
      value: 'Release'
    ${{ if eq(parameters.option, 'release-test')}}:
      value: 'Test'
  - name: theAgent
    value: ${{ parameters.agent }}
  - name: buildPlatform
    ${{ if eq(parameters.bits, '32') }}:
      value: 'x86'
    ${{ if eq(parameters.bits, '64') }}:
      value: 'x64'
  - name: nativePlatform
    ${{ if eq(parameters.bits, '32') }}:
      value: 'Win32'
    ${{ if eq(parameters.bits, '64') }}:
      value: 'x64'

jobs:
- ${{ if eq(parameters.option, 'cd') }}:
  - template: pipelines/cd-build.yml
- ${{ if eq(parameters.option, 'ci') }}:
  - template: pipelines/ci-build.yml
- ${{ if eq(parameters.option, 'release-build') }}:
  - template: pipelines/release-build.yml
- ${{ if eq(parameters.option, 'release-test') }}:
  - template: pipelines/release-test.yml

Contents of template pipelines/ci-build.yml. The other three templates are identical, except for the job name, down to the point that says "CI specific stuff starts here". Therefore, only one template is shown here:

# Template for CI build


jobs:
- job: ACM_CI_Build
  timeoutInMinutes: 0 # No time limit
  pool:
    name: 'Default'
    demands:
    - Agent.Name -equals $(theAgent)

  steps:
  - template: ./setup-tasks.yml

  - powershell: | 
        Copy-Item -Path "$(Pipeline.Workspace)\s\packages" -Destination "c:\users\public" -Force -Recurse 
    displayName: Save a copy of NuGet packages 

  - powershell: |
      Write-Host "CI specific stuff starts here"
    displayName: And so on

Contents of pipelines/setup_tasks.yml:

steps:
- checkout: self
  clean: true
  displayName: Checkout source 

- script: type c:\users\Public\dummy.txt
  displayName: 'Verify self-hosted agent'

- powershell: |
    Write-Host "This is where the problem becomes evident"
    Write-Host "... because the named files are not found their expected places."
    Write-Host "A CI job will exit with error."
    ForEach ( $file in (
      "$(Pipeline.Workspace)\s\Build\acm_components.targets",
      "$(Pipeline.Workspace)\s\Client\something\else.txt"
      )
    )
    {
      Write-Host "Doing stuff to $file"
    }
  displayName: The AAB hack

- script: |
    Write-Host "Rest of setup.yml is NuGet tasks, et cetera"
    Write-Host "By now, the damage is done"
  displayName: And so on

The problem is this: Of the four templates, three of them execute the checkout task the same way, and one of them (the ci one) does it differently.

Here's the relevant output from the three that are doing it the correct way.

Syncing repository: blabla/acm_source_821_azure_pipelines (Bitbucket)
Prepending Path environment variable with directory containing 'git.exe'.
...
##[command]git init "C:\acmbuild6\_work\1\s"
Initialized empty Git repository in C:/acmbuild6/_work/1/s/.git/
##[command]git remote add origin https://bitbucket.org/blabla/acm_source_821_azure_pipelines

Here's the way the ci task does it.

Syncing repository: blabla/acm_source_821_***_pipelines (Bitbucket)
Prepending Path environment variable with directory containing 'git.exe'.
##[command]git init "C:\acmbuild4\_work\2\s\acm_source_821_***_pipelines"
Initialized empty Git repository in C:/acmbuild4/_work/2/s/acm_source_821_***_pipelines/.git/
##[command]git remote add origin https://bitbucket.org/blabla/acm_source_821_***_pipelines

Why does the ADO checkout task use a different repository and a different target directory for my option ci than for my other options?

Where is this configuration info stored on my ADO VM (or perhaps in the repo), and how can I access it to edit the info?

Screenshot of the job flow, with resulting error.

Additional information that might help

• My agents are all self-hosted agents, running on an Azure VM.
• At the moment, I am triggering the builds manually, and manually setting the option value as a runtime variable.
• If I recall correctly, the ci option was the first option that I developed. That was a few months ago. The other three options came later. I only noticed this behavior recently, but it may have been there all along.

Troubleshooting work

I created a duplicate templte, called pipelines\ci-build2.yml. I copied the text from the original ci-build.yml template and pasted it into ci-build2.yml, instead of just running a command like cp ci-build.yml ci-build2.yml. That way I made sure I got only the text.

Then I created a new pipeline on ADO. This pipeline uses the same BBCloud repository as the original pipeline -- so its operation is identical to the original pipeline.

When I select a ci build on this new pipeline, it still uses the different repository, but this time it uses the correct target directory.

I don't know which of the two changes (different ci-build.yml template, different pipeline) fixed the problem, because I can't see what's happening under the hood. I don't know what parameters or config settings got changed. Can anyone give me some hints?

Answer 1

The issue was with a part of `pipelines/ci-build.yml' that I didn't put in the code listing in my original problem statement, because I didn't think it was relevant. It turned out to be the critical piece.

After completing the steps shown in the listing above, the next step for pipelines/ci-build.yml is to checkout the source a second time. The checkout tasks are identical in pipelines/setup-tasks.yml and pipelines/ci-build.yml.

- checkout: self
  clean: true

According to the Checkout path documentation, a pipeline can checkout a single repository or multiple repositories.

• If a single repository, it goes in c:\acmbuild6\_work\1\s. (following my pathnames from the original problem statement)
• If multiple repositories, each repository goes in its own subdirectory. So they go in c:\acmbuild6\_work\1\s\FirstRepoName and c:\acmbuild6\_work\1\s\SecondRepoName.

When I wrote the pipeline files, I assumed that since the same repository was being referenced in both files, it was counted as a "single repository". However, it appears that Azure Pipelines considers multiple checkout tasks to be "multiple repositories"; therefore, it checks out my code into c:\acmbuild6\_work\1\s\acm_source_821_azure_pipelines. Twice.

The simple solution to the problem is to specify the target path explicitly in the checkout task:

steps:
- checkout: self
  clean: true
  path: s

The path is defined relative to the pipeline workspace, c:\acmbuild6\_work\1 in my case, or c:\<agent>\_work\<n> in the general case. This solution has been tested, and it works.

A better solution is to rewrite either the pipeline or the project source, so that the second checkout in ci-build.yml is unnecessary.