Deadlock with async Task.Run method with Wait from synchronous method and timeout

Question

I have a method defined as:

public Task<ReturnsMessage> Function()
{
    var task = Task.Run(() =>
    {
         var result = SyncMethod();
         return new ReturnMessage(result);
    });
    
    if (task.Wait(delay))
    {
        return task;
    }

    var tcs = new TaskCompletionSource<ReturnMessage>();
    tcs.SetCanceled();
    return tcs.Task;
}

Now it is called in a loop based on maxAttempts value:

(method name RetryableInvoke)

for (var i = 0; i < maxAttempts; i++)
{
    try
    {
        return Function().Result;
    }
    catch (Exception e)
    {
    }
}

It works perfectly fine, however when there is a massive load I am finding that the threads are increasing drastically and dump is showing me this warnings:

Can anyone suggest me the best possible way to handle this situation, so that I don't see any kind of deadlocks?

Answer 1

You are not deadlocking the application. While Task.Run would execute the delegate on a background thread, the following call of Task.Wait effectively converts the originally concurrent Task.Run code back to synchronous code: Task.Wait and Task.Result synchronously wait for the Task to complete.

What you assume to be a deadlock is rather the result of a frozen main thread, caused by synchronously waiting for a long-running operation to complete.
Below the provided solutions is a section that briefly explains how Task.Result, Task.Wait and Task.GetAwaiter().GetResult() can create a real deadlock.

Although the Task related deadlock and your frozen main thread have different causes, the solution to both is similar: convert the synchronous code to an asynchronous version by introducing async/await. This will keep the main thread responsive.

Using async/await will replace the synchronous Task.Wait and also make Task.Result redundant.

The following two examples show how to convert your synchronous version to asynchronous code and provide two solutions to implement a timeout for a Task: the first and not recommended solution uses Task.WhenAny together with cancellation.
The second and recommended solution uses the timeout feature of the CancellationTokenSource.

It is generally recommended to check the class API for asynchronous versions instead of using Task.Run. In case your 3rd party library exposes an asynchronous API you should use this to replace the Task.Run. This is because true asynchronous implementations access the kernel to use the system's hardware to execute code asynchronously without the need to create background threads.

Timeout implementation using Task.WhenAny (not recommended)

This example uses the characteristic of Task.WhenAny which awaits a set of Task objects and returns the one that completes first and we usually cancel the remaining ones.
When creating a timed Task for example by using Task.Delay and pass it along with other Task objects to Task.WhenAny, we can create a race: if the timed task completes first, we can cancel the remaining Task objects.

public async Task<ReturnMessage> FunctionAsync()
{
  using var cancellationTokenSource = new CancellationTokenSource();
  {
    // Create the Task with cancellation support
    var task = Task.Run(
      () =>
      {
        // Check if the task needs to be cancelled
        // because the timeout task ran to completion first
        cancellationTokenSource.Token.ThrowIfCancellationRequested();

        // It is recommended to pass a CancellationToken to the
        // SyncMethod() too to allow more fine grained cancellation
        var result = SyncMethod(cancellationTokenSource.Token);
        return new ReturnMessage(result);
      }, cancellationTokenSource.Token);

    var timeout = TimeSpan.FromMilliseconds(500);

    // Create a timeout Task with cancellation support
    var timeoutTask = Task.Delay(timeout, cancellationTokenSource.Token);
    Task firstCompletedTask = await Task.WhenAny(task, timeoutTask);

    // Cancel the remaining Task that has lost the race.
    cancellationTokenSource.Cancel();

    if (firstCompletedTask == timeoutTask)
    {
      // The 'timoutTask' has won the race and has completed before the delay. 
      // Return an empty result.
      // Because the cancellation was triggered inside this class,
      // we can avoid to re-throw the OperationCanceledException
      // and return an error/empty result.
      return new ReturnMessage(null);
    }

    // The 'task' has won the race, therefore
    // return its result
    return await task;
  }
}

Timeout implementation using the CancellationTokenSouce constructor overload (recommended)

This example uses a specific constructor overload that accepts a TimeSpan to configure a timeout. When the defined timeout has expired the CancellationTokeSource will automatically cancel itself.

public async Task<ReturnsMessage> FunctionAsync()
{
  var timeout = TimeSpan.FromMilliseconds(500);
  using (var timeoutCancellationTokenSource = new CancellationTokenSource(timeout))
  {
    try
    {
      return await Task.Run(
        () =>
        {
          // Check if the timeout has elapsed
          timeoutCancellationTokenSource.Token.ThrowIfCancellationRequested();

          // Allow every called method to invoke the cancellation
          var result = SyncMethod(timeoutCancellationTokenSource.Token);
          return new ReturnMessage(result);
        }, timeoutCancellationTokenSource.Token);
    }
    catch (OperationCanceledException)
    {          
      // Return an empty result.
      // Because the cancellation was triggered inside this class,
      // we can avoid to re-throw the OperationCanceledException
      // and return an error/empty result.
      return new ReturnMessage(null);
    }
  }
}

To complete both of the above examples that converted the originally synchronous code (Function() to asynchronous code (FunctionAsync()), we have to await the new method properly:

// The caller of this new asynchronous version must be await this method too.
// `await` must be used  up the call tree whenever a method defined as `async`. 
public async Task<ReturnMessage> void RetryableInvokeAsync()
{ 
  ReturnMessage message = null;
  for (var i = 0; i < maxAttempts; i++)
  {
    message = await FunctionAsync();

    // Never use exceptions to control the flow.
    // Control the for-loop using a condition based on the result.
    if (!string.IsNullOrWhiteSpace(message.Text))
    {
      break;
    }
  }

  return message;
}

---

Why Task.Result, Task.Wait and Task.GetAwaiter().GetResult() create a deadlock when used with async code

First, because a method is defined using the async key word it supports asynchronous execution of operations with the help of the await operator. Common types that support await are Task, Task<TResult>, ValueTask, ValueTask<TResult> or any object that matches the criteria of an awaitable expression.

Inside the async method the await captures the SynchronizationContext of the calling thread, the thread it is executed on. await will also create a callback for the code that follows the await statement, called "continuation".

This callback is enqueued on the captured SynchronizationContext (by using SynchronizationContext.Post) and executed once the awaitable (e.g., Task) has signalled its completion.

Now that the callback is enqueued (stored) for later execution, await allows the current thread to continue to do work asynchronously while executing the awaitable.

async/await basically instructs the compiler to create a state machine.

Given is the following example that produces three potential deadlocks:

public void RetryableInvoke()
{ 
  // Potential deadlock #1.
  // Result forces the asynchronous method to be executed synchronously.
  string textMessage = FunctionAsync().Result;

  // Potential deadlock #2.
  // Wait forces the asynchronous method to be executed synchronously.
  // The caller is literally waiting for the Task to return.
  textMessage = FunctionAsync().Wait();

  // Potential deadlock #3.
  // Task.GetAwaiter().GetResult() forces the asynchronous method to be executed synchronously.
  // The caller is literally waiting for the Task to return.
  textMessage = FunctionAsync().GetAwaiter().GetResult();
}

private async Task<string> FunctionAsync() 
{
  // Capture the SynchronizationContext of the caller's thread by awaiting the Task.
  // Because the calling thread synchronously waits, the callers context is not available to process the continuation callback.
  // This means that the awaited Task can't complete and return to the captured context ==> deadlock
  string someText = await Task.Run(
    () =>
    {
      /* Background thread context */

      return "A message";
    });

  /* 
     Code after the await is the continuation context 
     and is executed on the captured SynchronizationContext of the 
     thread that called FunctionAsync. 
     
     In case ConfigureAwait is explicitly set to false, 
     the continuation is the same as the context of the background thread.
   */

  // The following lines can only be executed 
  // after the awaited Task has successfully returned (to the captured SynchronizationContext)
  someText += " More text";
  return someText;
}

In the above example Task.Run executes the delegate on a new ThreadPool thread. Because of the await the caller thread can continue to execute other operations instead of just waiting.

Once the Task has signaled its completed, the pending enqueued continuation callback is ready for execution on the captured SynchronizationContext.
This requires the caller thread to stop its current work to finish the remaining code that comes after the await statement by executing the continuation callback.

The above example breaks the asynchronous concept by using Task.Wait, Task.Result and Task.GetAwaiter().GetResult(). This Task members will synchronously wait and therefore effectively cause the Task to execute synchronously, which means:

a) The caller thread will block itself (wait) until the Task is completed. Because the thread is now synchronously waiting it is not able to do anything else. If instead the invocation had been asynchronously using async/await, then instead of waiting, the parent thread will continue to execute other operations (e.g., UI related operations in case of a UI thread) until the Task signals its completion.
Because of the synchronous waiting the caller thread can't execute the pending continuation callback.

b) The Task signals completion but can't return until the continuation callback has executed. The remaining code (the code after Task.Wait) is supposed to be executed on captured SynchronizationContext, which is the caller thread.
Since the caller thread is still synchronously waiting for the Task to return, it can't execute the pending continuation callback.
Now the Task must wait for the caller thread to be ready/responsive (finished with the synchronous waiting) to execute the continuation.

a) and b) describe the mutual exclusive situation that finally locks both the caller and the thread pool thread: the caller thread is waiting for the Task, and the Task is waiting for the caller thread. Both wait for each other indefinitely. This is the deadlock. If the caller thread is the main thread then the complete application is deadlocked and hangs.

Because the example used Task.Wait in one place and Task.Result in another, it has created two potential deadlock situations:

From Microsoft Docs:

Accessing the property's [Task.Result] get accessor blocks the calling thread until the asynchronous operation is complete; it is equivalent to calling the Wait method.

Task.GetAwaiter().GetResult() creates the third potential deadlock.

How to fix the deadlock

To fix the deadlock we can use async/await (recommend) or ConfigreAwait(false).

ConfigreAwait(true) is the implicit default: the continuation callback is always executed on the captured SynchronizationConext.

ConfigreAwait(false) instructs await (the state machine) to execute the continuation callback on the current thread pool thread of the awaited Task. It configures await to ignore the captured SynchronizationContext.

ConfigreAwait(false) basically wraps the original Task to create a replacement Task that causes await to not enqueue the continuation callback on the captured SyncronizationContext.

It's recommended to always use ConfigreAwait(false) where executing the callback on the caller thread is not required (this is the case for library code or non UI code in general). This is because ConfigreAwait(false) improves the performance of your code as it avoids the overhead introduced by registering the callback.
ConfigreAwait(false) should be used on the complete call tree (on every await not only the first).

Note, there are exceptions where the callback still executes on the caller's SynchronizationContext, for example when awaiting an already completed Task.

public async Task RetryableInvokeAsync()
{ 
  // Awaiting the Task always guarantees the caller thread to remain responsive. 
  // It can temporarily leave the context and will therefore not block it.
  string textMessage = await FunctionAsync();

  // **Only** if the awaited Task was configured 
  // to not execute the continuation on the caller's SnchronizationContext
  // by using ConfigureAwait(false), the caller can use 
  // Wait(), GetAwaiter().GetResult() or access the Result property 
  // without creating a deadlock
  string textMessage = FunctionAsync.Result;
}

private async Task<string> FunctionAsync() 
{
  // Because the awaited Task is configured to not use the caller's SynchronizationContext by using ConfigureAwait(false), 
  // the Task don't need to return to the captured context 
  // ==> no deadlock potential
  string someText = await Task.Run(
    () =>
    {
      /* Background thread context */

      return "A message";
    }).ConfigureAwait(false);

  /* 
     Code after the await is the continuation context 
     and is not executed on the captured SynchronizationContext.

     Because ConfigureAwait is explicitly set to false, 
     the continuation is the same as the context of the background thread.
    
     Additionally, ConfigureAwait(false) has improved the performance 
     of the async code. 
  */

  // The following lines will always execute
  // after the awaited Task has ran to completion
  someText += " More text";
  return someText;
}