XUnit and MSTest with ExpectedException Returning Different Results

Question

I have a project that is using a yield return and do not understand why XUnit is failing to catch an exception in my unit test while MSTest is passing.

Here is my dummy code.

The bizarre thing is that if I take my private method, EnumerableYieldReturn, and put that logic directly in my public method, YieldReturnList, the outcomes flip with the XUnit test passing and the MSTest failing.

[TestClass]
public class MSTestRunner
{
    [TestMethod]
    [ExpectedException(typeof(ArgumentException))]
    public void MSTestUsingExpectedException()
    {
        var sut = new YieldReturn();

        sut.YieldReturnList(null);
    }
}

public class XUnitRunner
{
    [Fact]
    [ExpectedException(typeof(ArgumentException))]
    public void XUnitUsingExpectedException()
    {
        var sut = new YieldReturn();

        sut.YieldReturnList(null);
    }
}

public class YieldReturn
{
    public IEnumerable<string> YieldReturnList(int? value)
    {
        if (value == null)
            throw new ArgumentException();

        return EnumerableYieldReturn((int)value);
    }

    private IEnumerable<string> EnumerableYieldReturn(int value)
    {
        var returnList = new List<string>() { "1", "2", "3" };

        for (int i = 0; i < value; i++)
        {
            yield return returnList[i];
        }
    }
}

I can get them both to pass by assigning the return object from sut.YieldReturnList and attempting to iterate through it but that doesn't explain why one framework is passing and the other is failing...

Answer 1

"xUnit.net has done away with the ExpectedException attribute in favor of Assert.Throws." from https://xunit.github.io/docs/comparisons.html.

The reason for the flip of results is that the exception is no longer thrown so:

MSTest: expects the exception because it uses the attribute, and therefore fails because it doesn't get the exception

XUnit: ignores the expect exception attribute because the framework doesn't use it, and therefore passes because an exception didn't cause the test to fail.

The reason the exception is no longer thrown if you change the methods is more complicated but it basically has to do with creating a state machine for methods that use the yield key word. Currently your public method doesn't use the yield key word directly so it is treated like a normal function and therefore executes the null check and throws the exception as soon as the method is called. Moving the yield key word to the public method makes it a lazy state machine so it doesn't execute the null check to throw the exception until you start to iterate the IEnumerable.

Also considering that your public method doesn't return a List you might consider renaming it.

here is how to correctly write the unit tests for the two frameworks

[TestClass]
public class MSTestRunner
{
    [TestMethod]
    [ExpectedException(typeof(ArgumentException))]
    public void MSTestUsingExpectedException()
    {
        var sut = new YieldReturn();

        sut.YieldReturnList(null).Any();
    }
}

public class XUnitRunner
{
    [Fact]
    public void XUnitUsingExpectedException()
    {
        var sut = new YieldReturn();

        Assert.Throws<ArgumentException>(() => sut.YieldReturnList(null).Any());
    }
}