How can I queue HttpClient requests until a Polly retry policy is successful?

Question

I'm using polly to refresh an authorisation token if a http request returns a 401 error. My full solution is based off: Refresh Token using Polly with Named Client, but the relevant parts are:

Retry policy:

static IAsyncPolicy<HttpResponseMessage> GetAuthRefreshPolicy(IServiceProvider serviceProvider)
{
        return Policy<HttpResponseMessage>
            .HandleResult(response => response.StatusCode == HttpStatusCode.Unauthorized)
            .RetryAsync(async (handler, retry) => await serviceProvider.GetRequiredService<IAuthorisationService>().RefreshTokenAsync());
}

AuthorisationService:

public class AuthorisationService : IAuthorisationService
{
    public String? AuthToken { get; private set; }

    private readonly IAuthenticationClient _AuthenticationClient;
    private readonly AuthOptions _AuthOptions;

    public AuthorisationService(IAuthenticationClient authenticationClient, IOptions<AuthOptions> options)
    {
        _AuthenticationClient = authenticationClient;
        _AuthOptions = options.Value;
    }

    public async Task RefreshTokenAsync()
    {
        this.AuthToken = await _AuthenticationClient.LoginAsync(_AuthOptions.User, _AuthOptions.Password);
    }
}

This works fine in most circumstances. The problem I am having is that if the auth token is currently invalid and multiple requests are made at the same time, because they are executed asynchronously they will all be called with the invalid token, fail and trigger the polly retry policy resulting in multiple unnecessary token refresh requests.

Is there a way to set polly up so that if a request fails with a 401, all subsequent requests will wait until the token has been successfully refreshed? Or to queue up retry attempts and skip requesting a new token if it's already been refreshed?

I have tried looking into polly features like bulkhead circuit breakers, but can't work out if these are right for this use case or if there's a simpler way to lock the token refreshing.

Answer 1

When I wrote the answers I thought it was clear that the code samples are not production-ready.

But lets focus on your specific question. In order to avoid multiple simultaneous concurrent token refresh calls you have to guard the TokenService's RefreshToken method. This can be done in many different ways. Let me show you how to do that with SemaphoreSlim:

public class TokenService : ITokenService
{
    private readonly SemaphoreSlim semaphore = new(1);
    private DateTime lastRefreshed = DateTime.UtcNow;
    public Token GetToken()
        => new() { Scheme = "Bearer", AccessToken = lastRefreshed.ToString("HH:mm:ss")}; 

    public async Task RefreshToken()
    {
        await semaphore.WaitAsync();

        try
        {
            //Use any arbitrary logic to detect simultaneous calls  
            if(lastRefreshed - DateTime.UtcNow < TimeSpan.FromSeconds(3)) 
            {
                Console.WriteLine("No refreshment happened");
                return;
            }
            
            Console.WriteLine("Refreshment happened");
            lastRefreshed = DateTime.UtcNow;
        }
        finally
        {
            semaphore.Release();
        }
    }
}

Let's test the code

var service = new TokenService();
for (int i = 0; i < 10; i++)
{
    Task.Run(async() =>
    {
        await Task.Delay(Random.Shared.Next() % 1000);
        await service.RefreshToken();
    });
}

The ouput will be:

Refreshment happened
No refreshment happened
No refreshment happened
No refreshment happened
No refreshment happened
No refreshment happened
No refreshment happened
No refreshment happened
No refreshment happened
No refreshment happened

---

UPDATE #1

As it was spotted in this github issue comment the above example does not work as intended.

After playing a bit with the code I've realized the root cause

if(lastRefreshed - DateTime.UtcNow < TimeSpan.FromSeconds(3))

This will be always true since it will be negative.

// Fix 1
if(DateTime.UtcNow - lastRefreshed < TimeSpan.FromSeconds(3))

// or Fix 2
if((lastRefreshed - DateTime.UtcNow).Duration() < TimeSpan.FromSeconds(1).Duration()) 

Dotnet fiddle