Why does a test method with DataTestMethod decorator fail to run more than once when multiple DataRows with array arguments are specified?

Question

I discovered surprising behaviour of the DataTestMethod and DataRow attributes. I have implemented the following test; of course the overall context is more complex, but this is a minimal example.

[TestClass]
public class SomeTest
{
    [DataRow(new int[] { 1, 2 })]
    [DataRow(new int[] { 3, 4 })]
    [DataRow(new int[] { 3, 5 })]
    [DataRow(new int[] { 3, 6 })]
    [DataTestMethod]
    public void ErrorStuff(int[] array)
    {
        Assert.IsNotNull(array);
    }
}

I was expecting that during test execution, ErrorStuff is executed four times (namely for every DataRow provided). However, this is not the case, it is executed once (for the first DataRow, which I inspected by debugging the test).

It goes without saying that the test is not meaningful at all and there are other ways to test the desired behaviour. However, I wonder how this behaviour can be explained. As presented, it it just a toy example; however, this potentially means that some test cases are never executed and there is hardly any way to notice this.

Any ideas how the behaviour can be explained? Any help is appreciated!

However, please don't suggest technically different alternatives like usage of DynamicData or formulation of single test methods for the single test cases - I am aware of that, but that is not my point of interest here.

Answer 1

Upgrade your MS Test Framework to Version 3.0+.

You can do so via nuget. At the time of writing, the current is 3.0.4:

dotnet add package MSTest.TestFramework --version 3.0.4

After I upgraded my framework, this works as expected (4 test runs occur):

You may also need to change your .csproj file to match the version number of the package:

<ItemGroup>
    <PackageReference Include="MSTest.TestAdapter" Version="3.0.4" />
    <PackageReference Include="MSTest.TestFramework" Version="3.0.4" />
</ItemGroup>

---

This seems to of been a known issue, and appears to have been fixed with the release of MSTest V3 according to the post. The linked GitHub issue also goes into details of the bug:

To give you a little more context and explain you the difference between what you see in VS and in command line. Basically, VS asks MSTest to discover tests, we then return a list and VS is using each test ID as a unique identifier for run/debug. We have a bug in 2.2.3+ of MSTest where IDs are broken for parameterized tests under some circumstances. This leads to VS mixing tests and behaving like all tests have the data of the 1st test.

Note that is seems like if you run this through a Continuous-Integration (such as Azure DevOps), it worked. The post also explains why it worked in that case:

As for the CI, it is using a different mechanism where it's not asking for discovery then reuse information to run tests but directly runs tests hence you don't have the issue on CI.