Polly does not timeout

Question

I am trying to get Polly to try again on timeout after 3 seconds and also when certain http codes are returned. However, it doesn't time out until after 100 seconds when the HttpClient times out.

Here is my code:

private static Polly.Wrap.AsyncPolicyWrap<HttpResponseMessage> GetPolicy()
{
    var timeoutPolicy = Policy.TimeoutAsync(3, Polly.Timeout.TimeoutStrategy.Optimistic);

    var retryPolicy = Policy
        .Handle<HttpRequestException>()
        .OrResult<HttpResponseMessage>(r =>
            r.StatusCode == HttpStatusCode.TooManyRequests ||
            r.StatusCode == HttpStatusCode.ServiceUnavailable ||
            r.StatusCode == HttpStatusCode.Forbidden)
        .WaitAndRetryAsync(3, i => TimeSpan.FromSeconds(3));

    var policy = retryPolicy.WrapAsync(timeoutPolicy);
    return policy;
}

Update

As requested, here is code where I am using the policy.

var pollyResponse = await GetPolicy().ExecuteAndCaptureAsync(() =>
      httpClient.SendAsync(GetMessage(HttpMethod.Delete, endpoint))
);

And the helper method that makes the HttpRequestMessage:

private HttpRequestMessage GetMessage<T>(HttpMethod method, string endpoint, T content)
{
    var message = new HttpRequestMessage
    {
        Method = method,
        RequestUri = new Uri(endpoint),
        Headers = {
                    { "MyCustomHeader", _value },
                    { HttpRequestHeader.Accept.ToString(), "application/json" }
                }
    };

    if (content != null)
    {
        var contentAsString = JsonSerializer.Serialize(content);
        message.Content = new StringContent(contentAsString);
    }

    return message;
}

Answer 1

First, let me share with you the revised version of your GetPolicy:

private static IAsyncPolicy<HttpResponseMessage> GetStrategy()
{
    var timeoutPolicy = Policy
        .TimeoutAsync<HttpResponseMessage>(3, TimeoutStrategy.Optimistic,
        onTimeoutAsync: (_, __, ___, ____) =>
        {
            Console.WriteLine("Timeout has occurred");
            return Task.CompletedTask;
        });

    var retryPolicy = Policy
        .Handle<HttpRequestException>()
        .Or<TimeoutRejectedException>()
        .OrResult<HttpResponseMessage>(r =>
            r.StatusCode == (HttpStatusCode)429 ||
            r.StatusCode == HttpStatusCode.ServiceUnavailable ||
            r.StatusCode == HttpStatusCode.Forbidden)
        .WaitAndRetryAsync(3, i => TimeSpan.FromSeconds(3),
        onRetryAsync: (_, __, ___) =>
        {
            Console.WriteLine("Retry will fire soon");
            return Task.CompletedTask;
        });

    return Policy.WrapAsync(retryPolicy, timeoutPolicy);
}

• I've changed the return type because from the consumer perspective the PolicyWrap is just an implementation detail
  • You could also use the AsyncPolicy<T> abstract class as return type if you don't want to use an interface (IAsyncPolicy<T>)
• I've added some debug logging (onTimeoutAsync, onRetryAsync) to be able to watch which policy triggers when
• I've added an Or<TimeoutRejectedException>() builder function call on the retryPolicy to make sure that retry will be triggered in case of timeout
  • Please note that the Timeout policy will throw its own exception
• I've also changed your retryPolicy.WrapAsync to a PolicyWrap because with that the escalation chain is more explicit
  • The left most policy is the most outer
  • The right most policy is the most inner
• I've also changed the timeoutPolicy (.TimeoutAsync < HttpResponseMessage > ) to align with the retry policy (both of them are wrapping a delegate which might return a Task<HttpResponseMessage>)

---

In order to be able to test our resilience strategy (note the naming) I've created the following helper method:

private static HttpClient client = new HttpClient();
public static async Task<HttpResponseMessage> CallOverloadedAPI(int responseDelay = 5000, int responseCode = 200)
{
    return await client.GetAsync($"http://httpstat.us/{responseCode}?sleep={responseDelay}");
}

• It will issue a request against a website which will return with a specified status code after a predefined amount of time
  • If you haven't used this website before please visit: 1, 2

Now, let's call the website:

public static async Task Main()
{
    HttpResponseMessage response;
    try
    {
        response = await GetStrategy().ExecuteAsync(async () => await CallOverloadedAPI());
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.Message);
        Environment.Exit(-1);
    }
    Console.WriteLine("Finished");
}

The output:

Finished

Wait, what??? The thing is none of the policies have been triggered.

Why? Because after 5 seconds we have received a response with 200.

But, we have set up a timeout, right? Yes and no. :) Even though we have defined a timeout policy we haven't really connected that to the HttpClient

So, how can I connect? Well, via CancellationToken

So, in case of timeout policy if a CancellationToken is in use then it can call its Cancel method to indicate the timeout fact to the HttpClient. And HttpClient will cancel the pending request.

Please note that, because we are using TimeoutPolicy the exception will be TimeoutRejectedException, not an OperationCanceledException.

---

So, let's modify our code to accept a CancellationToken

public static async Task<HttpResponseMessage> CallOverloadedAPI(int responseDelay = 5000, int responseCode = 200, CancellationToken token = default)
{
    return await client.GetAsync($"http://httpstat.us/{responseCode}?sleep={responseDelay}", token);
}

We have to adjust the usage side as well:

public static async Task Main()
{
    HttpResponseMessage response;
    try
    {
        response = await GetStrategy().ExecuteAsync(async (ct) => await CallOverloadedAPI(token: ct), CancellationToken.None);
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.Message);
        Environment.Exit(-1);
    }
    Console.WriteLine("Finished");
}

Now the output now will look like this:

Timeout has occurred
Retry will fire soon
Timeout has occurred
Retry will fire soon
Timeout has occurred
Retry will fire soon
Timeout has occurred
The delegate executed asynchronously through TimeoutPolicy did not complete within the timeout.

The last line is the Message of the TimeoutRejectedException.

---

Please note that if we remove the Or<TimeoutRejectedException>() call from the retryPolicy builder then the output will be the following:

Timeout has occurred
The delegate executed asynchronously through TimeoutPolicy did not complete within the timeout.

So, now retry will be triggered. There will be no escalation.

---

For the sake of completeness, here is the whole source code:

public static async Task Main()
{
    HttpResponseMessage response;
    try
    {
        response = await GetStrategy().ExecuteAsync(async (ct) => await CallOverloadedAPI(token: ct), CancellationToken.None);
    }
    catch (Exception ex)
    {
        Console.WriteLine(ex.Message);
        Environment.Exit(-1);
    }
    Console.WriteLine("Finished");
}

private static AsyncPolicy<HttpResponseMessage> GetStrategy()
{
    var timeoutPolicy = Policy
        .TimeoutAsync<HttpResponseMessage>(3, TimeoutStrategy.Optimistic,
        onTimeoutAsync: (_, __, ___, ____) =>
        {
            Console.WriteLine("Timeout has occurred");
            return Task.CompletedTask;
        });

    var retryPolicy = Policy
        .Handle<HttpRequestException>()
        .Or<TimeoutRejectedException>()
        .OrResult<HttpResponseMessage>(r =>
            r.StatusCode == (HttpStatusCode)429 ||
            r.StatusCode == HttpStatusCode.ServiceUnavailable ||
            r.StatusCode == HttpStatusCode.Forbidden)
        .WaitAndRetryAsync(3, i => TimeSpan.FromSeconds(3),
        onRetryAsync: (_, __, ___) =>
        {
            Console.WriteLine("Retry will fire soon");
            return Task.CompletedTask;
        });

    return Policy.WrapAsync(retryPolicy, timeoutPolicy);
}

private static HttpClient client = new HttpClient();
public static async Task<HttpResponseMessage> CallOverloadedAPI(int responseDelay = 5000, int responseCode = 200, CancellationToken token = default)
{
    return await client.GetAsync($"http://httpstat.us/{responseCode}?sleep={responseDelay}", token);
}