Taurus tests failing even if still inside success criteria

Question

I'm running tests in an Azure yaml pipeline with a jmx file and using Taurus for running and reporting results.

My pipeline has a parameter of success rate which is being passed like this (among other parameters):

parameters:
  SuccessRate: '97%'

And the part which is calling Taurus is defined like this and using this parameter looks like this:

- task: TaurusRunner@0
        displayName: 'Run Tests'
        inputs:
          taurusConfig: |
          ...
          ...
          reporting:
            - module: junit-xml
              filename: taurus-output/TEST-Taurus.xml
            - module: passfail
              criteria:
               - success<${{ parameters.SuccessRate }}, continue as failed
          ...

- task: PublishTestResults@2
        displayName: 'Publish Test Result'
          inputs:
            testResultsFormat: 'JUnit'
            testResultsFiles: 'taurus-output/TEST-Taurus.xml'
            failTaskOnFailedTests: true

The problem is that when I run the tests, there are some tests which are passing at 99.8%, yet the reporting still shows them as failed, even though I expected it to be considered as passed since the criteria if for 97%..

Why are the tests reported as not passing?

Answer 1

Based on the discussion in the comment, I can fully understand your situation now.

First of all, the TaurusRunner task has already return 0. This means that the Test has passed and Passfail module can work as expected.

The reason for Publish Test Result task failure is related to setting: failTaskOnFailedTests: true. This task will read the test output XML file and upload it. If there are failed tests in the file, the task will fail. It will not follow the Passfail criteria in Taurus Runner task. We can remove this argument in Publish Test Result task to let the task pass.

It does have exit code 0. Let's say I remove failtaskonfailedtest: true and all requests have 96%, then will it already fail in the run tests step?

I can reproduce the same situation.

The root cause of this problem is the different test statistics.

The Taurus Runner task will calculate based on the actual number of tests.

The Tests tab will calculate based on output xml(TEST-Taurus.xml). And the output xml will collect data based on test class(Another word: Request label).

Therefore, we see the difference in the success rate of the tests on both sides.

The Passfail module can only be used for Taurus Runner tasks, and will not affect subsequent Pipeline tasks(e.g. Publish test result task). Therefore, it will be counted based on the actual number of tests. In your case, the Taurus Test should be successful.

Also, is there a way to tell the passfail module to not only return code 0 if the criteria is met, but also give a success notice for the requests even if their success is below 100%?

When you set up the Passfail module, you can directly determine whether the test is successful based on the task running results.

When the exit code is 0(Test Pass), the task running result is Green. And the exit code is 3(Test fail), the task running result is Red.

Test Pass:

Test Fail: