c#dotnet-httpclient asp.net-core-2.1 polly retry-logic

Polly HandleTransientHttpError not catching HttpRequestException

I've created a retry policy on my HttpClient in the Startup.ConfigureServices method. Note also that by default, asp.net core 2.1 logs 4 [Information] lines for each call made by the HttpClient which are shows in the logs at the end of my question.

services.AddHttpClient("ResilientClient")
            .AddPolicyHandler(
                Policy.WrapAsync(
                    PollyRetryPolicies.TransientErrorRetryPolicy(),
                    Policy.TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(60))));

The policy is defined as follows. Note that I write the retry attempt to logs, so I will know if the retry policy is invoked.

public static IAsyncPolicy < HttpResponseMessage > TransientErrorRetryPolicy() {
    return HttpPolicyExtensions
        .HandleTransientHttpError()
        .Or < TimeoutRejectedException > ()
        .WaitAndRetryAsync(sleepDurations: ExponentialBackoffPolicy.DecorrelatedJitter(3, SEED_DELAY, MAX_DELAY),
            onRetry: (message, timespan, attempt, context) => {
                context.GetLogger() ? .LogInformation($ "Retrying request to {message?.Result?.RequestMessage?.RequestUri} in {timespan.TotalSeconds} seconds. Retry attempt {attempt}.");
    });
}

HandleTransientHttpError() is a Polly extension that states in it's comments:

The conditions configured to be handled are: • Network failures (as System.Net.Http.HttpRequestException)

My httpclient usage is like this:

using (HttpResponseMessage response = await _httpClient.SendAsync(request)) 
{
    response.EnsureSuccessStatusCode();

    try 
    {
        string result = await response.Content.ReadAsStringAsync();
        if (result == null || result.Trim().Length == 0) {
            result = "[]";
        }
        return JArray.Parse(result);
    } catch (Exception ex) {
        _logger.LogInformation($ "Failed to read response from {url}. {ex.GetType()}:{ex.Message}");
        throw new ActivityException($ "Failed to read response from {url}.", ex);
    }
}

The following logs are captured:

[Information] System.Net.Http.HttpClient.ResilientClient.LogicalHandler: Start processing HTTP request GET https://api.au.... obfuscated
[Information] System.Net.Http.HttpClient.ResilientClient.CustomClientHandler: Sending HTTP request GET https://api.au..... obfuscated
[Information] System.Net.Http.HttpClient.ResilientClient.CustomClientHandler: Received HTTP response after 2421.8895ms - 200
[Information] System.Net.Http.HttpClient.ResilientClient.LogicalHandler: End processing HTTP request after 2422.1636ms - OK
    
Unknown error responding to request: HttpRequestException:
System.Net.Http.HttpRequestException: Error while copying content to a stream. ---> System.IO.IOException: The server returned an invalid or unrecognized response.

at System.Net.Http.HttpConnection.FillAsync()
at System.Net.Http.HttpConnection.ChunkedEncodingReadStream.CopyToAsyncCore(Stream destination, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.HttpConnectionResponseContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
--- End of inner exception stack trace ---
at System.Net.Http.HttpContent.LoadIntoBufferAsyncCore(Task serializeToStreamTask, MemoryStream tempBuffer)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at nd_activity_service.Controllers.ActivityController.GetND(String url) in /codebuild/output/src251819872/src/src/nd-activity-service/Controllers/ActivityController.cs:line 561

The Http call succeeds, and I can see it returns 200 - OK. But then the HttpRequestException is thrown. I assume the policy is not being invoked because the HttpClient message pipeline has already resolved, as we can see it returned 200 - OK. So how is it throwing an exception outside of this?

And how do I handle it? Wrap another policy around the method that handles HttpRequestExceptions specifically?

This error does appear to be transient. It is a scheduled job and works the next time it is called.

Solution

Your policy is defined against the HttpClient not against the HttpResponseMessage.

So, the response.EnsureSuccessStatusCode() will not trigger retry even if you receive for example 428.

The HandleTransientHttpError will trigger retry if you receive 408 or 5XX status codes from the downstream system. And when the SendAsync throws the HttpRequestException

Because your exception StackTrace looks like this:

System.Net.Http.HttpRequestException: Error while copying content to a stream.

System.IO.IOException: The server returned an invalid or unrecognized response.

that's why my educated guess is that this exception is thrown by the HttpContent class while you try to read the response body (ReadAsStringAsync).

This will not trigger retry since you have defined your policy on the HttpClient.

If you want to retry in those cases as well when either the response.EnsureSuccessStatusCode() throws HRE or when the response.Content.ReadAsStringAsync() does then you have to wrap your whole http communication and response processing logic into a retry policy.

Let me show you how to do that.

First use a PolicyRegistry instead of AddPolicyHandler:

//services.AddHttpClient("ResilientClient")
//    .AddPolicyHandler(
//        Policy.WrapAsync(
//            TransientErrorRetryPolicy(),
//            Policy.TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(60))));

services.AddHttpClient("ResilientClient");
var registry = services.AddPolicyRegistry();
registry.Add("retry", Policy.WrapAsync(
            TransientErrorRetryPolicy(),
            Policy.TimeoutAsync<HttpResponseMessage>(TimeSpan.FromSeconds(60))));

Then ask the DI for the register, for example:

private readonly IHttpClientFactory factory;
private readonly IReadOnlyPolicyRegistry<string> registry;

public TestController(IHttpClientFactory factory, IReadOnlyPolicyRegistry<string> registry)
{
    this.factory = factory;
    this.registry = registry;
}

Finally retrieve the combined policy and execute the http call:

var retryPolicy = registry.Get<IAsyncPolicy<HttpResponseMessage>>("retry");
await retryPolicy.ExecuteAsync(async () => await IssueRequest());

private async Task<HttpResponseMessage> IssueRequest()
{
    var _httpClient = factory.CreateClient("ResilientClient");
    HttpResponseMessage response = await _httpClient.GetAsync("http://httpstat.us/428");

    response.EnsureSuccessStatusCode();
    return response;
}

I've used the httpstat.us to simulate 428 response.