Essentially, I have a pipeline, that sort of works how we expect it to. Everything works with a single runner, in a waterfall way.
Setup > Build > Unit tests > e2e tests
Then in 2nd stage, once all that passes, we publish the artefact.
Since Unit and e2e tests are in the same Stage, the reporter correctly logs them in the coverage section and shows e2e and unit tests together.
However, I've tried to split these in two jobs so two parallel runners can run to make the run faster.
The split is, Job 1:
Checks out repo, setup > unit tests > PublishCodeCoverageResults@2 Job 2:
Checks out repo, setup > build > e2e suite using Playwright > PublishTestResults@2 Once both these jobs succeed, an Artefact is generated.
These same tasks, when run in a single runner, generate the tests report correctly, however, in a parallel setup, they don't. Only the Playwright code coverage is displayed.
Here is the yaml Im currently working on:
trigger:
- main
- release/*
pool:
vmImage: ubuntu-latest
stages:
- stage: Validation
displayName: Validation
jobs:
- job: run_e2e_tests
displayName: Validate and Build
pool:
vmImage: ubuntu-latest
steps:
- task: Npm@1
displayName: Install bun
inputs:
command: custom
customCommand: install -g [email protected]
- script: |
bun install --frozen-lockfile
displayName: 'Install dependencies'
- script: bun run build
displayName: Build project
- script: bun run playwright install --with-deps
displayName: Setup playwright for e2e tests
# Continue even if the tests fail, so that the e2e tests can be run, and the test results can be published.
# The PublishTestResults task will fail if there are any failed tests.
continueOnError: true
- script: bun run e2e --reporter=junit
displayName: e2e tests
# Continue even if the tests fail, so that the test results can be published.
# The PublishTestResults task will fail if there are any failed tests.
continueOnError: true
env:
CI: true
PLAYWRIGHT_JUNIT_OUTPUT_NAME: junit-e2e.xml
- task: PublishTestResults@2
displayName: Publish test results
inputs:
testResultsFormat: JUnit
testResultsFiles: junit*.xml
failTaskOnFailedTests: true
failTaskOnFailureToPublishResults: true
failTaskOnMissingResultsFile: true
- job: run_unit_tests
displayName: Run Unit tests
pool:
vmImage: ubuntu-latest
steps:
- task: Npm@1
displayName: Install bun
inputs:
command: custom
customCommand: install -g [email protected]
- script: |
bun install --frozen-lockfile
displayName: 'Install dependencies'
- script: bun run test --coverage --reporters=jest-junit
displayName: Unit tests
# Continue even if the tests fail, so that the e2e tests can be run, and the test results can be published.
# The PublishTestResults task will fail if there are any failed tests.
continueOnError: true
- task: PublishCodeCoverageResults@2
displayName: Publish unit test code coverage
inputs:
summaryFileLocation: $(System.DefaultWorkingDirectory)/coverage/clover.xml
mergeTestResults: true
failIfCoverageEmpty: true
- stage: ArtefactGeneration
displayName: Artefact Generation
dependsOn: Validation
jobs:
- job: PublishSourceCode
displayName: Publish Source Code
steps:
- task: PublishPipelineArtifact@1
displayName: Publish Source Code Artefact
inputs:
targetPath: $(Pipeline.Workspace)
artifact: frontend-source-code
publishLocation: pipeline
Please note that installing deps only take 10 seconds, so I didn't find it necessary to share artefacts between jobs.
Here is the current yml for comparison:
trigger:
- main
- release/*
pool:
vmImage: ubuntu-latest
stages:
- stage: Validation
displayName: Validation
jobs:
- job: StaticValidationAndTests
displayName: Static Validation and Tests
steps:
- task: NodeTool@0
displayName: Install Node.js
inputs:
versionSource: spec
versionSpec: 20.x
- task: Npm@1
displayName: Install bun
inputs:
command: custom
customCommand: install -g [email protected]
- script: bun install --frozen-lockfile
displayName: Install dependencies
- script: bun run build
displayName: Build project
- script: bun run lint
displayName: Run linter
- script: bun run test --coverage --reporters=jest-junit
displayName: Unit tests
# Continue even if the tests fail, so that the e2e tests can be run, and the test results can be published.
# The PublishTestResults task will fail if there are any failed tests.
continueOnError: true
- task: PublishCodeCoverageResults@2
displayName: Publish unit test code coverage
inputs:
summaryFileLocation: $(System.DefaultWorkingDirectory)/coverage/clover.xml
failIfCoverageEmpty: true
- script: bun run playwright install --with-deps
displayName: Install playwright
enabled: false
- script: bun run e2e --reporter=junit
displayName: e2e tests
# Continue even if the tests fail, so that the test results can be published.
# The PublishTestResults task will fail if there are any failed tests.
continueOnError: true
enabled: false
env:
CI: true
PLAYWRIGHT_JUNIT_OUTPUT_NAME: junit-e2e.xml
- task: PublishTestResults@2
displayName: Publish test results
inputs:
testResultsFormat: JUnit
testResultsFiles: junit*.xml
mergeTestResults: true
failTaskOnFailedTests: true
failTaskOnFailureToPublishResults: true
failTaskOnMissingResultsFile: true
- stage: ArtefactGeneration
displayName: Artefact Generation
dependsOn: Validation
jobs:
- job: PublishSourceCode
displayName: Publish Source Code
steps:
- task: PublishPipelineArtifact@1
displayName: Publish Source Code Artefact
inputs:
targetPath: $(Pipeline.Workspace)
artifact: frontend-source-code
publishLocation: pipeline
Compared the original yaml and current one, if the Unit tests and e2e tests are running in sequence or separately in two agents, will have same output.
In this scenario, you can remove PublishTestResults@2 and PublishCodeCoverageResults@2 in both jobs, publish the output in the two jobs. Add a new job to download the output artifacts above, then use PublishTestResults@2 and PublishTestResults@2 to publish the test result and coverage.