Synchronize multiple unit test assemblies

Question

I have assembly Common with class UnitTestSemaphoreLocker:

public static class UnitTestSemaphoreLocker
{
    private static readonly SemaphoreSlim Semaphore = new SemaphoreSlim(1, 1);

    public static void Lock([NotNull] Action work)
    {
        if (work == null) throw new ArgumentNullException(nameof(work));

        Semaphore.Wait();
        try
        {
            work();
        }
        finally
        {
            Semaphore.Release();
        }
    }

    public static async Task LockAsync([NotNull] Func<Task> worker)
    {
        if(worker == null) throw new ArgumentNullException(nameof(worker));

        await Semaphore.WaitAsync();
        try
        {
            await worker();
        }
        finally
        {
            Semaphore.Release();
        }
    }
}

I have two unit test assemblies, lets say TestAssemblyA and TestAssemblyB

Unit tests from both assemblies use one resource, that support only one call per time. So i want yo use my UnitTestSemaphoreLocker to lock unit test and make them executing one by one:

[TestMethod]
public void ConstructionTest()
{
    UnitTestSemaphoreLocker.Lock(() =>
    {
       //test body
    });
}

It works perfectly fine for tests in one assembly, but seems like two of them creating own instances of my common static UnitTestSemaphoreLocker class.

Is there any way to sync unit test from two assemblies? Meanwhile part of them must use async/await.

Answer 1

According to the observed behavior, Visual Studio test runner (VSTest) executes multiple test assemblies in parallel. It launches multiple processes (at most one per CPU core). Every such process contains its own copy of CLR, each loading some AppDomain's and of course, a separate instance of the test assembly.

All this means that the static members are not shared, and also that synchronizarion doesn't work across multiple instances of CLR, because of SemaphoreSlim, which relies on CLR synchronization primitives.

I can think of two possible solutions:

Solution 1: synchronize across processes

Replace SemaphoreSlim with Semaphore. The latter can be initialized as a machine-wide named semaphore, allowing multiple processes to synchronize over same semaphore instance identified by a string name. See an example here. See also Semaphore and SemaphoreSlim for more info.

If only few of your tests require synchronization, it can be a good solution. However, if most or all of the tests must be synchronized, it makes no sense to run parallel test processes -- see solution #2.

Solution 2: disable parallel test processes

You can control number of parallel processes through <MaxCpuCount> element in the .runsettings file. Setting its value to 1 should disable the parallel processes:

<RunSettings>
  <RunConfiguration>
    <MaxCpuCount>1</MaxCpuCount>
  </RunConfiguration>
</RunSettings>

See Configure unit tests by using a .runsettings file for more details.

That is not to say that you cannot parallelize your tests. MSTest has a feature named "in-assembly parallel test execution" (IAP), which can be enabled by including the following assembly-level attribute (usually in AssemblyInfo.cs):

[assembly: Parallelize(Workers = 0, Scope = ExecutionScope.MethodLevel)]

In contrast to VSTest runner which spawns processes, the IAP spawns multiple threads, all sharing the same AppDomain. In this case, there is only one copy of the static members, and CLR-based synchronization with SemaphoreSlim works as well.

More info on IAP in this post: MSTest V2: in-assembly parallel test execution