Deadlock using async Task and SemaphoreSlim

Question

we are running an ASP.NET 6 webapplication and are having strange issues with deadlocks. The app suddenly freezes after some weeks of operations and it seems that it might be caused by our locking mechanism with the SemaphoreSlim class.

I tried to reproduce the issue with a simple test-project and found something strange. The following code is simply starting 1000 tasks where each is doing some work (requesting semaphore-handle, waiting for 10 ms and releasing the semaphore). I expected this code to simply execute one task after another. But it freezes because of a deadlock in the first call of the DoWork method (at await Task.Delay(10)).

Does anyone know why this causes a deadlock? I tried exactly the same code with ThreadPool.QueueUserWorkItem instead of Task.Run and Thread.Sleep instead of Task.Delay and this worked as expected. But as soon as I use the tasks it stops working.

Here is the complete code-snippet:

internal class Program
{
    static int timeoutSec = 60;
    static SemaphoreSlim semaphore = new SemaphoreSlim(1);
    static int numPerIteration = 1000;
    static int iteration = 0;
    static int doneCounter = numPerIteration;
    static int successCount = 0;
    static int failedCount = 0;
    static Stopwatch sw = new Stopwatch();
    static Random rnd = new Random();

    static void Main(string[] args)
    {
        Task.WaitAll(TestUsingTasks());
    }

    static async Task TestUsingTasks()
    {
        while (true)
        {
            var tasks = new List<Task>();
            if (doneCounter >= numPerIteration)
            {
                doneCounter = 0;

                if (iteration >= 1)
                {
                    Log($"+++++ FINISHED TASK ITERATION {iteration} - SUCCESS: {successCount} - FAILURES: {failedCount} - Seconds: {sw.Elapsed.TotalSeconds:F1}", ConsoleColor.Magenta);
                }
                iteration++;

                sw.Restart();
                for (int i = 0; i < numPerIteration; i++)
                {
                    // Start indepdent tasks to do some work
                    Task.Run(async () =>
                    {
                        if (await DoWork())
                        {
                            successCount++;
                        }
                        else
                        {
                            failedCount++;
                        }
                        doneCounter++;
                    });
                }
            }
            await Task.Delay(10);
        }
    }

    static async Task<bool> DoWork()
    {
        if (semaphore.Wait(timeoutSec * 1000)) // Request the semaphore to ensure that one 1 task at a time can enter
        {
            Log($"Got handle for {iteration} within {sw.Elapsed.TotalSeconds:F1}", ConsoleColor.Green);
            var totalSec = sw.Elapsed.TotalSeconds;
            await Task.Delay(10); // Wait for 10ms to simulate some work => Deadlock seems to happen here
            Log($"RELEASING LOCK handle for {iteration} within {sw.Elapsed.TotalSeconds:F1}. WAIT took " + (sw.Elapsed.TotalSeconds - totalSec) + " seconds", ConsoleColor.Gray);
            semaphore.Release();
            return true;
        }
        else
        {
            Log($"ERROR: TASK handle failed for {iteration} within {sw.Elapsed.TotalSeconds:F1} sec", ConsoleColor.Red);
            return false;
        }
    }

    static void Log(string message, ConsoleColor color)
    {
        Console.ForegroundColor = color;
        Console.WriteLine(message);
        Console.ForegroundColor = ConsoleColor.White;
    }
}

Thanks in advance!

Answer 1

But it freezes because of a deadlock in the first call of the DoWork method (at await Task.Delay(10)).

I would argue that it is not deadlock but a thread starvation issue. If you wait long enough you will see that threads will be able to finish the simulation wait from time to time.

The quick fix here is using non-blocking WaitAsync call with await:

static async Task<bool> DoWork()
{
    if (await semaphore.WaitAsync(timeoutSec * 1000))
    {
         ...
    }
}

Also note:

• It is recommended to wrap the code after Wait.. into try-finally block and release the semaphore in the finally.
• Incrementing counters in parallel environments better should be done in atomic fashion, for example with Interlocked.Increment.