Search code examples
azure-devopsazure-pipelinestfvcappveyor

Adding and updating tests within Azure DevOps Pipeline programmatically using API

I'm converting one of our projects to use Azure DevOps rather than AppVeyor for CI. As part of the build we use a custom test runner to perform certain tests.

When running in AppVeyor we call the REST API directly from the test runner to inform the build server of the tests being run and to update their status. This is quite straightforward, as shown in the REST sections of their Add Tests and Update Tests documentation, and gives us great integration into AppVeyor's UI.

I've been researching how to do the same in Azure DevOps. I've found a section of the REST API for adding and updating test results. It isn't entirely clear from the API documentation if this is what I would use during a running pipeline, or if this is for some other scenario. I've searched for others trying to do the same thing, but without any luck so far. Most examples talk about uploading a test results file, but this seems a rather indirect way of getting the test results published, especially as I would like to register all the tests before they are run and then update their status as they are completed.

Does anyone have any pointers or examples around publishing test results using the Azure DevOps API during a build?

Solution in detail:

With Merlin Liang's answer for guidance I've now got this working.

Step 1. First I created a new Test Run, which I did at the end of my build job (called Build) using a powershell script:

  - powershell: |
      $url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/test/runs?api-version=5.0"
      $body = @{
          "name" = "IntegrationTests"
          "build" = @{
              "id" = $env:BUILD_BUILDID
          }
          "isAutomated" = $TRUE
          "state" = "InProgress"
      }
      $json = $body | ConvertTo-Json
      $result = Invoke-RestMethod -Method 'Post' -Uri $url -Body $json -ContentType 'application/json' -Headers @{
          Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
      }
      $runId = $result.id
      echo "##vso[task.setvariable variable=INTEGRATION_TEST_RUN_ID;isOutput=true]$runId"
    displayName: 'Create test run'
    name: CreateTestRun
    env:
      SYSTEM_ACCESSTOKEN: $(System.AccessToken)

Note you have to explicitly give the script access to the System.AccessToken in the env section, as that variable isn't put into an environment variable by default. The test run state should be InProgress to make sure the tests show in the build's test list while they are running.

Step 2. Next my pipeline fans out into parallel jobs to run the tests, which need access to the INTEGRATION_TEST_RUN_ID variable I created above, so I import that into the job:

  - job: IntegrationTests
    dependsOn: Build
    variables:
      INTEGRATION_TEST_RUN_ID: $[ dependencies.Build.outputs['CreateTestRun.INTEGRATION_TEST_RUN_ID'] ]
    strategy:
      parallel: 2
    ...

My test runner (written in C#) can then construct the URL for POSTing and PATCHing tests:

$"{this.apiUrl}{this.projectName}/_apis/test/runs/{this.testRunId}/results?api-version=5.0"

And adding test attachments:

$"{this.apiUrl}{this.projectName}/_apis/test/runs/{this.testRunId}/results/{testResultId}/attachments?api-version=5.0-preview.1"

Where those class member variables are just read from the usual environment variables, and testResultId is the ID returned by the API when I initially POST the test.

Again, I'm explicitly setting the SYSTEM_ACCESSTOKEN in an env section when I call the test runner from the Azure Pipeline YAML, and using it in the authorization header.

Step 3. Finally I fan-in again mark the test run as complete in a dedicated job which depends on the integration test jobs and must always run even if tests fail:

  - job: EndIntegrationTests
    dependsOn:
      - Build
      - IntegrationTests
    condition: always()
    variables:
      INTEGRATION_TEST_RUN_ID: $[ dependencies.Build.outputs['CreateTestRun.INTEGRATION_TEST_RUN_ID'] ]
    steps:
    - checkout: none

    - powershell: |
        $url = "$($env:SYSTEM_TEAMFOUNDATIONCOLLECTIONURI)$env:SYSTEM_TEAMPROJECTID/_apis/test/runs/$($env:INTEGRATION_TEST_RUN_ID)?api-version=5.0"
        Write-Host "URL: $url"
        $body = @{
            "state" = "Completed"
        }
        $json = $body | ConvertTo-Json
        Invoke-RestMethod -Method 'Patch' -Uri $url -Body $json -ContentType 'application/json' -Headers @{
            Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
        }
      displayName: 'Complete test run'
      env:
        SYSTEM_ACCESSTOKEN: $(System.AccessToken)

Note that this job needs to explicitly depend on Build so that I can read the output variable INTEGRATION_TEST_RUN_ID, and it obviously depends on IntegrationTests so that it won't complete the test run until all the tests are finished.

Solution

  • You are very close to the answer. Please see this doc: Surface test results in the Tests tab.

    Except uploading test results via tasks, such as Publish Test Results task, VS test or etc. We also provide a way to upload the results via Rest api:

    enter image description here

    To achieve what you want, you must create a test run first with one known build number, then post test results to that.

    1. Create a new test to existing build, specify the buildID in request body.

    Sample:

    POST https://dev.azure.com/{org name}/{project}/_apis/test/Runs/{runId}/results?api-version=5.0

{
 "name": "NewTest", 
 "build": { "id": "162" }, 
 "isAutomated": true, 
 "state": "Waiting" 
}

    1. Run test, so that the test result could become available.

    2. Now, post this new test result into the test which created previously.

    [ { "testCaseTitle": "ReferBuild", "automatedTestName": "ReferBuildAuto", "priority": 1, "outcome": "Passed" }]

    Here you could refer to the example which showed in the doc.

    1. At last, you can use update api.