Why does NUnit not start executing the next test when a thread is suspended?

Question

I would like to know why NUnit does not start the next async test when await Task.Delay is called.

I have a test project where each test submits a job to a service in the cloud, polls for the results, and then asserts the results that come back. The cloud service can run thousands of jobs in parallel. In production, a job can take up to a couple hours to run, but the test jobs I am submitting each run in less than a minute.

Currently up to 10 tests run in a parallel, but each test synchronously waits on the results to come back before the next test can start.

I am trying to think of way to make all tests complete faster (I understand about unit tests. We have though as well, but these tests have a different purpose). One idea is to use the async/await functionality built into .Net to suspend the thread one test is running on and start the next test while the first test is waiting on results from the cloud service. I built a little test project to see if this idea would work.

These tests were written on .Net framework v4.7.2. Using Nunit 3.13.2.

using NUnit.Framework;
using System;
using System.Collections.Concurrent;
using System.IO;
using System.Threading.Tasks;

namespace TestNUnit
{
  [TestFixture]
  public class Test
  {
    private static ConcurrentQueue<string> _queue = new ConcurrentQueue<string>();

    [Test]
    public async Task Test1()
    {
      _queue.Enqueue($"Starting test 1 {DateTime.Now}");
      await Task.Delay(12000);
      Assert.IsTrue(true);
      _queue.Enqueue($"  Ending test 1 {DateTime.Now}");
    }

    [Test]
    public async Task Test2()
    {
      _queue.Enqueue($"Starting test 2 {DateTime.Now}");
      await Task.Delay(10000);
      Assert.IsTrue(true);
      _queue.Enqueue($"  Ending test 2 {DateTime.Now}");
    }


    [Test]
    public async Task Test3()
    {
      _queue.Enqueue($"Starting test 3 {DateTime.Now}");
      await Task.Delay(14000);
      Assert.IsTrue(true);
      _queue.Enqueue($"  Ending test 3 {DateTime.Now}");
    }

    [OneTimeTearDown]
    public void Cleanup()
    {
      File.AppendAllLines("C:\\temp\\nunit.txt", _queue.ToArray() );
    }
  }
}

In my AssemblyInfo file I added the following lines to run 2 tests in parallel.

[assembly: Parallelizable(ParallelScope.All)]
[assembly: LevelOfParallelism(2)]

I hoped to see that all 3 tests started before any tests finished. Instead what I see is that 2 tests start and the third test starts after 1 of the other two tests finish. I tried running in both Visual Studio and the command line using the NUnitLite nuget package.

I tried the same test on XUnit and MsTest v2. When running the command line version of XUnit, I saw the behavior I wanted to see. Everywhere else though I see that 1 test has to finish before the third test starts.

Does anyone know why this would only work when running the command line version of XUnit? Thanks!

Here is the output when the test framework waits for 1 test to finish before starting the third test:

Starting test 2 3/4/2022 7:40:18 AM
Starting test 1 3/4/2022 7:40:18 AM
  Ending test 2 3/4/2022 7:40:28 AM
Starting test 3 3/4/2022 7:40:28 AM
  Ending test 1 3/4/2022 7:40:30 AM
  Ending test 3 3/4/2022 7:40:42 AM

Here is the output when all three tests start before any finishes:

Starting test 1 3/3/2022 4:19:09 PM
Starting test 3 3/3/2022 4:19:09 PM
Starting test 2 3/3/2022 4:19:09 PM
  Ending test 2 3/3/2022 4:19:19 PM
  Ending test 1 3/3/2022 4:19:21 PM
  Ending test 3 3/3/2022 4:19:23 PM

Answer 1

NUnit (and most likely the other framework runners where this didn't work for you) sets up the number of threads you specify to run tests. So, when you reach a point where all those tests have entered a wait state, no further tests can be run.

The assumption in this design is that wait states are rare and short, which is almost always true for unit tests, which was the original intended application for most of these runners - most certainly for nunitlite.

For a runner to continue after all the originally allocated threads were waiting, it would have to be re-designed to create new threads dynamically.

UPDATE:

Responding to the comment about the ThreadPool... The NUnit parallel dispatcher doesn't use the ThreadPool. At the time I wrote it, the ThreadPool wasn't available on all platforms we supported, so each TestWorker creates its own thread. If it were written today, it might well use the ThreadPool as you imagined.

I'm afraid that this may answer the question of your title without actually helping you. But it does suggest a workaround...

Since the only threads you will ever get are those initially allocated, you may simply increase the number of threads. For example... if you estimate that your particular set of tests will be in a wait state 50% of the time, use 20 threads rather than 10. Experiment with the number until you hit the best level for your workload. If the ideal number turns out (as is likely) to be different on your desktop versus your CI/CD environment, then provide it as an option to the command rather than using the attribute.