I have some integration tests that go out to an external service and may occasionally fail. For these, I'm running 'Assert.Inconclusive("errMsgHere");' so that the test doesn't cause the whole CI/CD pipeline to fail; I just want a warning. While this does give a warning icon when I'm running the tests locally in Visual Studio, for Azure DevOps the test is reported as 'not run'. Also, in the pipeline run's 'Tests' tab, the test's Debug details have a blank entry for the error message instead of "errMsgHere", even though that error message is included in the outputted .trx file! Surely Azure DevOps should show the message outputted by Assert.Inconclusive(). Is there any way to get at this message, or get the test result reported as something more useful than "not run"? Perhaps 'warning' or 'inconclusive'?
Per this response from Sarabjot Singh on the github issue for this task, it appears to be this behavior by design.
@osolo The Inconclusive tests are treated the same as skipped tests.
The behavior for not reporting them as failed and failing the task is by design currently. It is based on the precedence set by the test platform (https://github.com/microsoft/vstest) and the test framework (https://github.com/microsoft/testfx)
What is the behavior when you run the tests locally on your machine using the vstest.console runner? You can generate the trx result file (using --logger:trx option) and then compare it what is being shown in the pipeline UI.
Additionally, the linked bug filed on the develop community seems unlikely to be resolved:
I have detected that for the last 35 days, this issue didn't have much product team activity and a very small amount of new votes or comments. Based on this, its severity, and affected area, it’s my experience that this issue is very unlikely to be fixed.
There does seem to be a workaround mentioned, although it might not meet your desired outcome. I know it probably is not what you want, but you can treat failures on the tests as warnings by setting the "Continue on error" option for the test task. It would treat all failures as warnings though, so maybe there were cases you actually did want to stop the pipeline would not be useful for your case.