Search code examples
c#xunitopencoverappveyorcoveralls

AppVeyor + OpenCover + xUnit - Build doesn't fail when tests fail

I created an AppVeyor Build Script that uses OpenCover and Coveralls.Net to run my xUnit tests and publish the code coverage to Coveralls.io.

But when I have failing tests - AppVeyor reports the build successful. How do I configure AppVeyor to fail if OpenCover + xUnit report failing tests?

The script is based on csMACnz's sample:

.\src\packages\OpenCover.4.6.519\tools\OpenCover.Console.exe 
   -register:user 
   -target:"xunit.console.clr4.exe"  
   "-targetargs:""src\HttpWebRequestWrapper.Tests\bin\$env:CONFIGURATION\HttpWebRequestWrapper.Tests.dll"" 
   /noshadow 
   /appveyor" 
   -filter:"+[HttpWebRequestWrapper*]*" 
   -output:opencoverCoverage.xml

    $coveralls = (Resolve-Path "src/packages/coveralls.net.*/tools/csmacnz.coveralls.exe").ToString()
    & $coveralls --opencover -i opencoverCoverage.xml --repoToken $env:COVERALLS_REPO_TOKEN --commitId $env:APPVEYOR_REPO_COMMIT --commitBranch $env:APPVEYOR_REPO_BRANCH --commitAuthor $env:APPVEYOR_REPO_COMMIT_AUTHOR --commitEmail $env:APPVEYOR_REPO_COMMIT_AUTHOR_EMAIL --commitMessage $env:APPVEYOR_REPO_COMMIT_MESSAGE --jobId $env:APPVEYOR_JOB_ID

I've tried adding the -returntargetcode flag to the OpenCover.Console.exe code, but that doesn't seem to signal AppVeyor to fail the build.

Solution

  • I think the reason is simple. Following how you wrote the YML file, AppVeyor expects the whole test_script to provide a non-zero return code, while a failure inside the PowerShell script section you left won't lead to such a case.

    You have to run each commands separately, and then any failure will fail the whole build, as my AppVeyor script shows

    test_script:
  - dotnet test Tests\CSharpCore\Tests.NetStandard.csproj --configuration Release --no-build --filter "FullyQualifiedName~Lextm.SharpSnmpLib.Integration.TrapDaemonTestFixture"
  - dotnet test Tests\CSharpCore\Tests.NetStandard.csproj --configuration Release --no-build --filter "FullyQualifiedName~Lextm.SharpSnmpLib.Integration.DaemonTestFixture"
  - dotnet test Tests\CSharpCore\Tests.NetStandard.csproj --configuration Release --no-build --filter "FullyQualifiedName~Lextm.SharpSnmpLib.Unit"

    If I wrap the three commands in a batch file or PowerShell script, I can reproduce the same issue you experienced.