Execute Xunit tests in theory sequentially (not parallel)

Question

We host configuration files for different environements in our git repo. As part of the CI process I'd like to makesure that these config files are always valid. For this I've created this test which copies the configurations, tries to start the server and shuts it down right away.

public class DeployConfigurationValidationTests
{
    #region Private Fields

    private readonly ITestOutputHelper _testOutputHelper;
    private const string ServerBaseUrl = "http://localhost:44315";

    #endregion

    #region Constructors

    public DeployConfigurationValidationTests(ITestOutputHelper testOutputHelper)
    {
        _testOutputHelper = testOutputHelper;
    }

    #endregion

    #region Public Tests

    /// <summary>
    /// Copies all files contained in the directory specified by parameter <see cref="deployConfigDirectoryPath"/> to the executing directory and launches the application with this configuration.
    /// </summary>
    /// <param name="deployConfigDirectoryPath">The path of the directory containing the deploy configurations</param>
    [Theory]
    [InlineData("../../../../../Configurations/Dev/")]
    [InlineData("../../../../../Configurations/Int/")]
    [InlineData("../../../../../Configurations/Prod/")]
    public async Task ValidateDeployConfigurationsTest(string deployConfigDirectoryPath)
    {
        // Arrange (copy deploy configurations into directory where the test is running)
        var currentDirectory = Directory.GetCurrentDirectory();
        var configurationFilePaths = Directory.GetFiles(deployConfigDirectoryPath);
        foreach (var configurationFilePath in configurationFilePaths)
        {
            var configurationFileName = Path.GetFileName(configurationFilePath);
            var destinationFilePath = Path.Combine(currentDirectory, configurationFileName);
            File.Copy(configurationFilePath, Path.Combine(currentDirectory, destinationFilePath), true);
            _testOutputHelper.WriteLine($"Copied file '{Path.GetFullPath(configurationFilePath)}' to '{destinationFilePath}'");
        }

        // Act (launch the application with the deploy config)
        var hostBuilder = Program.CreateHostBuilder(null)
                                 .ConfigureWebHostDefaults(webHostBuilder =>
                                                           {
                                                               webHostBuilder.UseUrls(ServerBaseUrl);
                                                               webHostBuilder.UseTestServer();
                                                           });

        using var host = await hostBuilder.StartAsync();

        // Assert
        // Nothing to assert, if no error occurs, the config is fine
    }

    #endregion
}

The test is working fine when running each InlineData individually but fails when running the theory because the tests are being ran parallel by default. It will obviously not work to launch multiple (test) servers on the same port, using the same DLLs.

Question: How do I tell xUnit to run those tests sequentially?

We're using .net core 3.1 with XUnit 2.4.1

Answer 1

One way to solve this problem is to make use the CollectionAttribute.

Unfortunately you can apply this attribute only for classes.

So, you would need a small refactoring like this:

internal class ValidateDeploymentConfigBase
{
    public async Task ValidateDeployConfigurationsTest(string deployConfigDirectoryPath)
    {
        // Arrange
        var currentDirectory = Directory.GetCurrentDirectory();
        var configurationFilePaths = Directory.GetFiles(deployConfigDirectoryPath);
        foreach (var configurationFilePath in configurationFilePaths)
        {
            var configurationFileName = Path.GetFileName(configurationFilePath);
            var destinationFilePath = Path.Combine(currentDirectory, configurationFileName);
            File.Copy(configurationFilePath, Path.Combine(currentDirectory, destinationFilePath), true);
            _testOutputHelper.WriteLine($"Copied file '{Path.GetFullPath(configurationFilePath)}' to '{destinationFilePath}'");
        }


        var hostBuilder = Program.CreateHostBuilder(null)
                                    .ConfigureWebHostDefaults(webHostBuilder =>
                                    {
                                        webHostBuilder.UseUrls(ServerBaseUrl);
                                        webHostBuilder.UseTestServer();
                                    });

        // Act
        using var host = await hostBuilder.StartAsync();
    }
}

And then your test cases would look like this:

[Collection("Sequential")]
internal class ValidateDevDeploymentConfig: ValidateDeploymentConfigBase
{
    [Fact]
    public async Task ValidateDeployConfigurationsTest(string deployConfigDirectoryPath)
    {
        base.ValidateDeployConfigurationsTest("../../../../../Configurations/Dev/");
    }
}

...

[Collection("Sequential")]
internal class ValidateProdDeploymentConfig : ValidateDeploymentConfigBase
{
    [Fact]
    public async Task ValidateDeployConfigurationsTest(string deployConfigDirectoryPath)
    {
        base.ValidateDeployConfigurationsTest("../../../../../Configurations/Prod/");
    }
}