Cannot make Polly Timeout Policy override the HttpClient default timeout

Question

I am using a Polly retry policy and as expected during that retry processes the HttpClient hits its 100 second timeout. I have tried a couple of different ways to incorporate the Polly Timeout policy to move the timeout to per retry instead of total, but the 100 second timeout keeps firing.

I have read about 5 StackOverflow questions that all say to wrap the policies, and I even found the demo on the Polly wiki that say to call AddPolicyHandler twice. That did not work either. I am sure I have some very basic wrong. Please explain to me the error of my ways.

I am using .net core 5 preview 7 and current Polly nuget packages.

Typed HttpClient registration

serviceCollection.AddHttpClient<APIWrappers.SendGridClientEmail, APIWrappers.SendGridClientEmail>((c) =>
{
    c.BaseAddress = new Uri("https://api.sendgrid.com/v3a/");
})
    .AddPolicyHandler((services, req) => TransientLogAndRetry<APIWrappers.SendGridClientEmail>(services, req)
    .WrapAsync(Policy.TimeoutAsync<HttpResponseMessage>(10)));

The Policy definition

public IAsyncPolicy<HttpResponseMessage> TransientLogAndRetry<T>(IServiceProvider services, object req)
{
    Console.WriteLine($"Name: {typeof(T).Name} - Time: {DateTime.Now}");
    return HttpPolicyExtensions.HandleTransientHttpError().
        OrResult((rsp) => rsp.StatusCode == System.Net.HttpStatusCode.TooManyRequests).
        Or<Polly.Timeout.TimeoutRejectedException>().
        WaitAndRetryAsync(5, GetRetryTime,
#pragma warning disable CS1998
            async (ex, ts, rc, ctx) =>
            {
                ((STS.Diagnostics.LoggerFactory)services.GetService<ILoggerFactory>())?.CreateLogger<T>().
                    Warn(this, "HTTP request above will retry in {0}", ts);
            }
#pragma warning restore CS1998
        );
}

The delay / penalty calculation

private TimeSpan GetRetryTime(int retryCount, DelegateResult<HttpResponseMessage> response, Context context)
{
    TimeSpan result;
    long unix;

    Console.WriteLine($"Count: {retryCount} - Time: {DateTime.Now}");
    if (response.Result is null || response.Result.StatusCode != System.Net.HttpStatusCode.TooManyRequests)
        result = TimeSpan.FromSeconds(Math.Min(Math.Pow(Math.Sqrt(retryCount * 2), 5), 60 * 60));
    else
    {
        unix = response.Result.Headers.GetValues("X-RateLimit-Reset").Select((v) => long.Parse(v)).First();
        result = DateTime.UtcNow - DateTimeOffset.FromUnixTimeSeconds(unix);
    }
    Console.WriteLine(result);
    return result;
}

Answer 1

Based on the provided code I can see the following resilient strategy:

100 seconds global timeout

5 retries (= 5+1 attempts)

Exponential back-off logic

10 seconds local timeout

---

If I understand your problem correctly then you are looking for a solution to increase the global timeout.

The simplest solution (1) is to increase the HttpClient instance's Timeout property from the default 100 to the desired value. If this timeout is exceeded then the instance will throw an OperationCanceledException and no further retry attempt can be established.

If the HttpClient instance is shared / reused amongst different components then you might need to change the global timeout per usage. You can overcome on this by introducing a new Timeout policy between the HttpClient's global Timeout and your Retry policy.

So, you could end up with the following setup:

200 seconds global timeout (by HttpClient

150 seconds overall / overarching timeout (by outer timeout policy)

5 retries (= 5+1 attempts)

Exponential back-off logic

10 seconds local timeout

In this particular case I would recommend to use Policy.WrapAsync (1) just like this:

Policy.WrapAsync(OverallTimeoutPolicy(), RetryPolicy(), LocalTimeoutPolicy());