Playwright test is failing in azure pipeline script and pipeline is not switching to next one

Question

My testing pipeline is splitted to severeal scripts, every script has from 1 to 4 tests inside. When script is passed pipeline starts execute next one. But if script fails, it informs about fail and is not moving futher, just counting time till agent job stops

Here is pipeline yml:

# Node.js with React
# Build a Node.js project that uses React.
# Add steps that analyze code, save build artifacts, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/javascript



schedules:
- cron: "0 0 * * *"
  displayName: Daily midnight build
  branches:
    include:
    - master
  always: true

# variables:
#   CI: true

trigger:
- master

pool:
  name: Pool3

jobs:
- job: Test
  timeoutInMinutes: 180
  cancelTimeoutInMinutes: 2

  steps:
  - task: NodeTool@0
    inputs:
      versionSpec: '15.x'
    displayName: 'Install Node.js'
    condition: 


  - task: DownloadSecureFile@1
    inputs:
      secureFile: '.env'

  # - script: |
  #     npm ci
  #   displayName: "Install dependencies"

  - script: |
      npm install
      npx playwright install --with-deps
    displayName: "Install Playwright Browsers"

  - script: |
      npx playwright test 1.EndowmentTotalSurrenderAndClaimEventBank.spec.ts
    displayName: "Run 1.EndowmentTotalSurrenderAndClaimEventBank"

  - script: |
      npx playwright test 13.EndowmentPartialBank.spec.ts
    displayName: "Run 13.EndowmentPartialBank"

  - script: |
      npx playwright test 2.EdowmentTotalSurrenderAndClaimEventEnterOtherRequestTotal.spec.ts 6b.EndowmentClaimEventUnsettlement.spec.ts 7.EndowmentTotalSurrenderAndClaimEventBankOtherOutpayment.spec.ts 
    displayName: "Run EdowmentAndClaimEvent tests"

# 3.EndowmentTotalSurrenderAndClaimEventNetting.spec.ts 4.EndowmentClaimEventBankAndNetting.spec.ts

  - script: |
      npx playwright test 6a.EndowmentClaimEventBankCancelPayoutWithOutpaymentClaimWorkflow.spec.ts 11.EndowmentTotalSurrenderAndClaimEventBankCancelStateTaxes.spec.ts 
    displayName: "Run EndowmentCancel tests"

# 5.EndowmentClaimEventBankAndNettingCancel.spec.ts 12.EndowmentTotalSurrenderAndClaimEventNettingCancelStateTaxes.spec.ts

  - script: |
      npx playwright test 9.EndowmentTotalSurrenderAndClaimEventBankStateTaxes.spec.ts 
    displayName: "Run EndowmentTaxes tests"

# 10.EndowmentTotalSurrenderAndClaimEventNettingStateTaxes.spec.ts

  - script: |
      npx playwright test 8.Non-insuredClaimEvent.spec.ts 14.EndowmentTotalBank.spec.ts 15.EndowmentPolicyTermiantionPayoutTotalApproved.spec.ts
    displayName: "Run EndowmentOthers tests"

  - script: |
      npx playwright test 16.ManualPayableOutpaymentGroup.spec.ts 17.ManualPayableOutpaymentSplit.spec.ts 18.ManualPayableOutpaymentBank.spec.ts 19.ManualPayableOutpaymentOther.spec.ts 20.ManualPayableOutpaymentCanceled.spec.ts 
    displayName: "Run ManualBank tests"   
    
  - script: |
      npx playwright test 22.RefundCoverUncover.spec.ts 23.RefundGroup.spec.ts 24.RefundOtherOutpayment.spec.ts 25.RefundNetting.spec.ts
    displayName: "Run Refund tests"

  - script: |
      npx playwright test 21.ManualNetting.spec.ts 26.ManualNettingRemove.spec.ts 27.ManualPayableOutpaymentSplitBank.spec.ts 28.ManualPayableOutpaymentSplitUnsplit.spec.ts
    displayName: "Run ManualNetting tests"  

Here is result screenshot: enter image description here

Then all tests are passed in one script, it closes sucessufully and returning passed/failed tests. Is there problem in yml file?

Answer 1

Just try to enable the variable CI=true in your pipeline.

Alternately add "set CI=true" before running the tests.

  - script: |
      set CI=true && npx playwright test 22.RefundCoverUncover.spec.ts 23.RefundGroup.spec.ts 24.RefundOtherOutpayment.spec.ts 25.RefundNetting.spec.ts
    displayName: "Run Refund tests"

Update:

In your scenario, if a test script failed in a step, then it will skip next scripts, the step will be failed. And the pipeline will be considered as failed, so the next steps will also be skipped, the pipeline will be jumped to the end.

If you want to run all the test scripts in the pipeline, you can split the test scripts into separate steps (a step a script). Then set continueOnError: true for each step task. So even if the previous test step fails, it continues to the next step.

For example:

  - script: | 
      set CI=true && npx playwright test 22.RefundCoverUncover.spec.ts 
    displayName: "Run Refund test22"
    continueOnError: true

  - script: |
      set CI=true && npx playwright test 23.RefundGroup.spec.ts 
    displayName: "Run Refund test23"
    continueOnError: true

  - script: |
      set CI=true && npx playwright test 24.RefundOtherOutpayment.spec.ts
    displayName: "Run Refund test24"
    continueOnError: true
  
  - script: |
      set CI=true && npx playwright test 25.RefundNetting.spec.ts
    displayName: "Run Refund test25"
    continueOnError: true