How can we read the Azure DevOps task log?

Question

I have a task in an azure pipeline that conducts some dynamic tests but since the tests are multiple in number, whether the individual test passes or fails, that is determined by whats there in the logs.

As such the Azure DevOps steps always turns green. I was looking to solve a problem by reading that log and then deciding what to do next based on whether the logs contain failure or not. So let's say, if the task turns green and logs detect failure, I will want to take a decision to not publish an artifact.

How can I solve this problem in a standard way by reading the logs of a previous task using a standard azure api in the next task and then taking a decision?

Clarification: as needed

These tests are not units, they are actually some custom java tests which are triggered via calling a .sh file from a self-hosted agent on a linux machine.

Answer 1

Edit: after clarification, I split my comment into two parts: your custom test case, and then the rest of my comment which was previously here, applicable for cases where standard unit testing frameworks are used.

Alternative 1: Solution possibility for custom testing, based on analyzing test log files

First you will need a script, e.g. PowerShell script, which interprets the log files, determines if tests have failed or not, and sets vso task.complete results property accordingly (see example below).

Let's call this file evaluate-test-results.ps1, and let's assume it interprets some sort of testResults.json data, containing an important property named haveTestsPassed, deciding whether or not the tests passed:

evaluate-test-results.ps1 file content:

<#
.SYNOPSIS
Checks test results.
#>

param (
    [Parameter(Mandatory = $true)]
    [string]$testResultsJson
)

try {
    $testResults = $testResultsJson | ConvertFrom-Json

    if($testResults.haveTestsPassed)
    {
       Write-Host "##vso[task.complete result=Succeeded;]Tests have successfully passed"
    }
    else
    {
       Write-Host "##vso[task.complete result=Failed;]Tests have NOT passed"
    }
}
catch {
    Write-Host "##vso[task.complete result=Failed;]Error while parsing test results from JSON log file..."
}

Then, you can use this script in a PowerShell@2 task, like this:

- task: PowerShell@2
  displayName: "PowerShell: Evaluate test results from testResults.json"
  inputs:
    targetType: filePath
    filePath: 'evaluate-test-results.ps1'
    arguments: "-testResultsJson 'testResults.json'"
    failOnStderr: true

Note:

You can also throw standard error when haveTestsPassed is false, alternatively, thanks to the failOnStderr property. However, this solution is a bit more official above.

Alternative 2: Solution possibilities for standard unit test frameworks

Simpler solution if dotnet test command is configured to run your tests

The simple solution would be like this, if your tests are bound to the dotnet test command, using for example xUnit. In such a scenario, this task will fail by default if one of your unit tests fail, without ever needing PublishTestResults@2 task:

# This task will run the 'dotnet test' command, as if you would from a CLI
- task: DotNetCoreCLI@2
  displayName: Run and publish tests
  inputs:
    command: 'test'

If you can't use dotnet test command

In this case, you have to use PublishTestResults@2 task to interpret test results. Set your testing framework/tool and test results file accordingly. Be sure to set failTaskOnFailedTests: true

- task: PublishTestResults@2
  inputs:
    testRunner: VSTest # or else...
    testResultsFiles: '**/*.trx' # or else...
    failTaskOnFailedTests: true