IHttpClient Polly Timeout and WaitAndRetry policy when handling concurrent http requests Clarification

Question

Just have a question about Pollys timeout/retry policy and how it works when handling concurrent http requests.

From reading/my own understanding the timeout and retry policy will be applied to each individual http request, so if we have 5 http requests, each of them would have there own timeout and retry policy, so from the code below each http request would timeout after 5 seconds and retry a total of 4 times.

    public override void Configure(IFunctionsHostBuilder builder)
    {
        
        var timeout = Policy.TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(5));

        builder.Services
            .AddHttpClient("PointsbetClient")
            .AddPolicyHandler(GetRetryPolicy())
            .AddPolicyHandler(timeout);
    }

    private static IAsyncPolicy<HttpResponseMessage> GetRetryPolicy()
    {
        return HttpPolicyExtensions
            .HandleTransientHttpError()
            .Or<TimeoutRejectedException>()
            .WaitAndRetryAsync(Backoff.DecorrelatedJitterBackoffV2(
                medianFirstRetryDelay: TimeSpan.FromMilliseconds(500),
                retryCount: 4));
    }

Now say I have a 1000 http requests to which I need to make GetAsync() calls to so I can scrape their data and for performance purposes, I am making these calls concurrently, by using await Task.WhenAll(tasks);. Since 1000 is way too many requests to hit at the one time I am using SemaphoreSlim class and limiting the MaxParallelRequests to 100.

How would the Polly retry and timeout policy apply now? does it still apply to each of those individual 1000 requests or will it treat 1 task containing 100 requests as a single timeout/retry policy? From my understanding it would still treat and apply policy to each of the individual http requests, I've been searching but can't find confirmation on this.

Answer 1

The short answer is yes they are treated separately.

---

In order to understand how the system works we have to look under the hood. Let's start our journey at the AddPolicyHandler.

Disclaimer: I've slightly edited the code snippets for the sake of brevity.

public static IHttpClientBuilder AddPolicyHandler(this IHttpClientBuilder builder, IAsyncPolicy<HttpResponseMessage> policy)
{
    if (builder == null) throw new ArgumentNullException(nameof(builder));
    if (policy == null) throw new ArgumentNullException(nameof(policy));
    builder.AddHttpMessageHandler(() => new PolicyHttpMessageHandler(policy));
    return builder;
}

This method is defined inside the PollyHttpClientBuilderExtensions class and it provides extension methods for the IHttpClientBuilder.

As you can see it does nothing else just registers yet another HttpMessageHandler into the chain.

Now, let's see how does this special handler look like

protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
{
    if (request == null) throw new ArgumentNullException(nameof(request));

    // Guarantee the existence of a context for every policy execution, 
    // but only create a new one if needed.
    // This allows later handlers to flow state if desired.
    var cleanUpContext = false;
    var context = request.GetPolicyExecutionContext();
    if (context == null)
    {
        context = new Context();
        request.SetPolicyExecutionContext(context);
        cleanUpContext = true;
    }

    HttpResponseMessage response;
    try
    {
        var policy = _policy ?? SelectPolicy(request);
        response = await policy.ExecuteAsync((c, ct) => SendCoreAsync(request, c, ct), context, cancellationToken).ConfigureAwait(false);
    }
    finally
    {
        if (cleanUpContext)
            request.SetPolicyExecutionContext(null);
    }

    return response;
}

This method is defined inside the PolicyHttpMessageHandler.

As you can see nothing extraordinary happens here

• We either retrieve or create a new Context
• We either retrieve the policy from registry or use the provided one
• We execute the policy which decorates the SendCoreAsync

So, where does the magic happen? Let's jump to the documentation comment of this class

All policies provided by Polly are designed to be efficient when used in a long-lived way. Certain policies such as the Bulkhead and Circuit-Breaker maintain state and should be scoped across calls you wish to share the Bulkhead or Circuit-Breaker state. Take care to ensure the correct lifetimes when using policies and message handlers together in custom scenarios. The extension methods provided by PollyHttpClientBuilderExtensions are designed to assign a long lifetime to policies and ensure that they can be used when the handler rotation feature is active.

To understand how does Retry differ from Circuit Breaker lets look at their Engines' Implementation signature

RetryEngine

internal static class RetryEngine
{
    internal static TResult Implementation<TResult>(
        Func<Context, CancellationToken, TResult> action,
        Context context,
        CancellationToken cancellationToken,
        ExceptionPredicates shouldRetryExceptionPredicates,
        ResultPredicates<TResult> shouldRetryResultPredicates,
        Action<DelegateResult<TResult>, TimeSpan, int, Context> onRetry,
        int permittedRetryCount = Int32.MaxValue,
        IEnumerable<TimeSpan> sleepDurationsEnumerable = null,
        Func<int, DelegateResult<TResult>, Context, TimeSpan> sleepDurationProvider = null)
    {
    ...
    }
}

CircuitBreakerEngine

internal class CircuitBreakerEngine
{
    internal static TResult Implementation<TResult>(
        Func<Context, CancellationToken, TResult> action,
        Context context,
        CancellationToken cancellationToken,
        ExceptionPredicates shouldHandleExceptionPredicates, 
        ResultPredicates<TResult> shouldHandleResultPredicates, 
        ICircuitController<TResult> breakerController)
    {
    ...
    }
}

What you have to spot here is the ICircuitController. The CircuitStateController base class stores the state information. One of its derived class is shared between the different policy executions

internal readonly ICircuitController<EmptyStruct> _breakerController;
...
CircuitBreakerEngine.Implementation(
    action,
    context,
    cancellationToken,
    ExceptionPredicates,
    ResultPredicates,
    _breakerController);

---

I hope this clarify things.