How to fix failed tests count in teamcity for re-executed tests?

Question

We are using TeamCity for Selenium tests execution with Specflow+Specrun, the problem is that TeamCity count re-executed tests.

For example if some test failed for first time he will be re-executed two times more, in teamcity we will see that three tests failed, but probably it was one test.

Also if first re-execution will fail, but other two will success, this will be reported in teamcity as two failed, one passed, but I need to be reported that only one test has been passed.

It is possible to configure this in TeamCity using service messages or something else?

UPDATED:

Based on answer we can gather logs using powershell script and change build status using teamcity service messages:

$path = Get-ChildItem %teamcity.build.checkoutDir%\ProjectFolder\bin\Remote\TestResults\specrun.log
$file = Get-content $path
$total = $file | select-string "Total:"
$passed = $file | select-string "Succeeded:"
$failed = $file | select-string "Failed:"

write-host $( "##teamcity[buildStatus text='Tests {0}, {1}, {2}']" -f $total, $passed, $failed )

Answer 1

TeamCity is behaving as expected. TeamCity reports what Testing Framework tells him. Roughly the process is:

• Execute step that run tests.
• Gather the output logs.
• Extract stats: number of test executed, passed and failing.
• Fail the build if tests fail if configured to do that.

You have to configure your Testing Framework (specflow+specrun in your case) to change the way it reports the re-execution. You should configure specflow to log the re-executions in a different way... if it can.

In general you should avoid re-execution of tests. The tests should work at the first attempt. They should prepare the state if needed, do their test, and clean the state.

The need of re-execution indicates probably some dependency with the other tests. Something like:

• Test 001: Search for user Foo1. (FAILS)
• Test 002: Adds user Foo1. (OK)
• Re-execution of Test 001: Search for user Foo1. (OK)

The correct way is that TEST001 does not have a dependency of TEST002, adds the user Foo1, test search and clean user Foo1.