Azure DevOps Pipeline: Tests that pass locally fail on the pipeline

Question

Our team is trying to implement Azure DevOps pipeline but we are having some issues. For background, we are running the tests on our self-hosted windows server. We got the front-end tests to all work but now we are having issues getting the backend to pass.

While the majority of the tests are passing just fine we have a few instances where it is not working as expected. Although they should be able to run in parallel, I turned that off to verify that wasn't the issue. I've also tried running them in isolation without any noticeable change.

The errors we get are just simple assert failures like the one below that don't give us much information.

 Error Message:
   Assert.AreEqual failed. Expected:<True>. Actual:<False>.

But the line numbers point to a mock setup like the one below which I don't understand.

mockQueryHandler.Setup(x => x.Handle(It.IsAny<FindQuery>())).Returns(info);

Currently the test class that is giving us the most problems is one that tests our event alert functionality. It seems that the first test will pass but the subsequent ones fail. However, we have the pipeline setup to rerun any failed tests three times until it gives up. So on every rerun the next test will pass. The only thing I can think of is that our auto mocks are giving us trouble for some reason. We set them up in accordance with AutoMock - How to unit test with Keyed Registrations?

We mock it out as follows:

  var IAbstractEventFactoryMock = new Mock<IAbstractEventFactory>();
  using (var mock = AutoMock.GetLoose(builder => builder.RegisterInstance(IAbstractEventFactoryMock.Object).Keyed<IAbstractEventFactory>("EventLogFactory").Keyed<IAbstractEventFactory>("ArpEventLogFactory"))) {
...
}

Below is our .runsettings file

<RunSettings>  
  <RunConfiguration>
    <ResultsDirectory>.\TestResults</ResultsDirectory>
  </RunConfiguration>

  <MSTest>
  </MSTest>
</RunSettings>

Below is the YAML we are using to run the tests

steps:
- task: VSTest@2
  displayName: 'Run All Tests'
  inputs:
    testAssemblyVer2: |
     **\*ID_Test*.dll
     !**\*TestAdapter.dll
     !**\obj\**
    searchFolder: '$(System.DefaultWorkingDirectory)/ID_Test'
    runSettingsFile: 'ID_Test/.runsettings'
    runInParallel: false
    runTestsInIsolation: true
    codeCoverageEnabled: false
    testRunTitle: 'Unit Tests'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    diagnosticsEnabled: true
    rerunFailedTests: true
  continueOnError: true

Again, these tests all pass without problems locally and they pass eventually if I run them enough times on the pipeline. So I don't think the problem is in the code itself. It seems to be that the tests aren't getting setup or cleaned properly. I am primarily looking for any ideas on other ways I can configure my yaml or runsettings to try to fix this issue. Thank you in advance for any help and let me know if I can provide any additional information to help.

Answer 1

Turned out this was just an issue with our tests. In a few places we would use the current time which was fine locally, but the server had a much higher execution time that created a few errors. Testing the pipeline on an agent that had a similar execution time as running the tests locally helped me discover that it wasn't an issue with the pipeline.