Failing unit tests result in obscure error message on CI build / pull request when using DotNetCoreCLI

Question

I've got our CI pipeline set up to run tests, but when a test fails, rather than getting a meaningful error like "2 unit tests failed", we get the following:

Error: The process 'C:\Program Files\dotnet\dotnet.exe' failed with exit code 1

Dotnet command failed with non-zero exit code on the following projects : [ '' ]

(It's worth nothing that the "projects" list is apparently empty in the second error when auto-detecting the solution file. If I explicitly specify the solution or projects path, it appears in the error)

Devs would have to go poring through the job console logs to see that actually they just broke a unit test, rather than it being something wrong with the DevOps infrastructure or CI pipeline setup (which is their assumption):

Here's how the solution is building and running tests:

steps:
- task: DotNetCoreCLI@2
  displayName: 'Restore NuGet Packages'
  inputs:
    command: 'restore'

- task: DotNetCoreCLI@2
  displayName: 'Build Projects'
  inputs:
    command: 'build'
    arguments: '--configuration $(buildConfiguration) --no-restore'
    
- task: DotNetCoreCLI@2
  displayName: 'Run Tests'
  inputs:
    command: 'test'
    arguments: '--configuration $(buildConfiguration) --no-build --no-restore'

- task: DotNetCoreCLI@2
  displayName: 'Publish Deployable Projects'
  inputs:
    command: 'publish'
    etc...

Is something lacking in this setup, or is the DotNetCoreCLI@2 "test" command just not mature in this regard?

I've used other pipelines using vstest / mstest commands directly and had nicer output, but I was hoping I could do everything with DotNetCoreCLI@2 for a variety of reasons, such as being able to chain several steps together without having to re-build the solution.

Answer 1

As a workaround, I've added a new pipeline step which runs only if the tests failed, and adds a new error output.

For whatever reason, pull request check failures only relay the error messages of the most recently failed task, so by adding a task which adds a new failure message, it "hides" the 2 obscure error messages, and replaces them with a more meaningful one:

Upon clicking the error (which navigates to the failed build) - we will still see the two obscure error messages, but also now the third one which the developers can hopefully focus on:

Here's the additions to the pipeline which take care of this:

- task: PowerShell@2
  displayName: 'Mark Build as Succeeded'
  inputs:
    targetType: 'inline'
    script: 'echo "##vso[task.setvariable variable=task.build_suceeded]true"'
  condition: succeeded()

- task: DotNetCoreCLI@2
  displayName: 'Run Tests'
  inputs:
    command: 'test'
    projects: '$(solution)'
    arguments: '--no-build --no-restore --configuration $(buildConfiguration) --collect "Code Coverage"'

- task: PowerShell@2
  displayName: 'Automated Tests Failed'
  condition: and (failed(), eq(variables['task.build_suceeded'], 'true'))
  inputs:
    targetType: 'inline'
    script: 'echo "##vso[task.logissue type=error]The build succeeded, but one or more tests failed. Check the `"Tests`" tab for details on failed tests."'

I went the extra mile here of setting a variable first to indicate that all prior steps succeeded (e.g. the restore / build steps) so that we only show this new error message when test test job fails, but everything else succeeds. If the build itself fails, this message won't show up.