I have an ASP.NET Core 3.1 Web API service that is receiving a web request, doing some manipulations to it, and then passing it on to a backend service and returning the response synchronously. It was working fine but I wanted to introduce some retry logic to those backend requests in case some blips occurred.
I'm using a typed HttpClient and attempting to use Polly to implement the retry logic: https://github.com/App-vNext/Polly/wiki/Polly-and-HttpClientFactory#using-polly-with-ihttpclientfactory
When the backend service works, everything seems to be fine but unfortunately, whenever my backend returns an error like a 500 Internal Server Error, I get the following exception:
Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware: Error: An unhandled exception has occurred while executing the request.
System.Net.Http.HttpRequestException: An error occurred while sending the request.
---> System.InvalidOperationException: The stream was already consumed. It cannot be read again.
at System.Net.Http.StreamContent.PrepareContent()
at System.Net.Http.StreamContent.SerializeToStreamAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
at System.Net.Http.HttpContent.CopyToAsync(Stream stream, TransportContext context, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.SendRequestContentAsync(HttpRequestMessage request, HttpContentWriteStream stream, CancellationToken cancellationToken)
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.HttpConnection.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithNtConnectionAuthAsync(HttpConnection connection, HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Polly.Retry.AsyncRetryEngine.ImplementationAsync[TResult](Func`3 action, Context context, CancellationToken cancellationToken, ExceptionPredicates shouldRetryExceptionPredicates, ResultPredicates`1 shouldRetryResultPredicates, Func`5 onRetryAsync, Int32 permittedRetryCount, IEnumerable`1 sleepDurationsEnumerable, Func`4 sleepDurationProvider, Boolean continueOnCapturedContext)
at Polly.AsyncPolicy`1.ExecuteAsync(Func`3 action, Context context, CancellationToken cancellationToken, Boolean continueOnCapturedContext)
at Microsoft.Extensions.Http.PolicyHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at Microsoft.Extensions.Http.Logging.LoggingScopeHttpMessageHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
at MyProject.MyController.Routing.HttpMessageRouter.SendRequestAndTrackTiming(Func`1 action, String destinationID) in /mnt/c/Users/myCode/source/repos/MyProject-GitLab/MyController/src/MyController/Routing/HttpMessageRouter.cs:line 59
at MyProject.MyController.Routing.HttpMessageRouter.SendNewRequest(IMessageWrapper`1 message) in /mnt/c/Users/myCode/source/repos/MyProject-GitLab/MyController/src/MyController/Routing/HttpMessageRouter.cs:line 33
at MyProject.MyController.Controllers.MyControllerController.Resource(String destinationId) in /mnt/c/Users/myCode/source/repos/MyProject-GitLab/MyController/src/MyController/Controllers/MyControllerController.cs:line 151
at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.TaskOfIActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeActionMethodAsync>g__Awaited|12_0(ControllerActionInvoker invoker, ValueTask`1 actionResultValueTask)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeNextActionFilterAsync>g__Awaited|10_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Rethrow(ActionExecutedContextSealed context)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.<InvokeInnerFilterAsync>g__Awaited|13_0(ControllerActionInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|19_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)
at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Logged|17_1(ResourceInvoker invoker)
at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)
at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)
at Prometheus.HttpMetrics.HttpRequestDurationMiddleware.Invoke(HttpContext context)
at Prometheus.HttpMetrics.HttpRequestCountMiddleware.Invoke(HttpContext context)
at Prometheus.HttpMetrics.HttpInProgressMiddleware.Invoke(HttpContext context)
at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)
The code I have to send the request is as follows. This first method actually just utilizes a delegate so I can transparently add some metrics gathering around the typed HttpClient call. The typed HttpClient is called RoutingClient in this code:
public async Task<IActionResult> SendNewRequest(IMessageWrapper<HttpRequestMessage> message)
{
HttpResponseMessage destinationResponse = await SendRequestAndTrackTiming(() => _client.SendAsync(message.Message), message.DestinationID);
return CreateSerializeableResponseMessage(destinationResponse);
}
private ResponseMessageResult CreateSerializeableResponseMessage(HttpResponseMessage httpResponse)
{
ResponseMessageResult responseMessage = new ResponseMessageResult(httpResponse);
IOutputFormatter[] formattersList = { new HttpResponseMessageOutputFormatter() };
FormatterCollection<IOutputFormatter> formattersCollection = new FormatterCollection<IOutputFormatter>(formattersList);
responseMessage.Formatters = formattersCollection;
return responseMessage;
}
private async Task<HttpResponseMessage> SendRequestAndTrackTiming(Func<Task<HttpResponseMessage>> action, string destinationID)
{
HttpResponseMessage response = null;
Stopwatch stopwatch = new Stopwatch();
try
{
stopwatch.Start();
response = await action();
return response;
}
finally
{
stopwatch.Stop();
HttpStatusCode statusCode = (response != null) ? response.StatusCode : HttpStatusCode.InternalServerError;
_routedMessageMetricTracker.Histogram.Observe(destinationID, statusCode, stopwatch.Elapsed.TotalSeconds);
}
}
This is my code in Startup.ConfigureServices() (actually this is in an extension method I defined):
public static IServiceCollection AddRoutingClient(this IServiceCollection services, RoutingSettings routingSettings)
{
List<TimeSpan> retryTimeSpans = new List<TimeSpan>();
// routingSettings.RetrySeconds is just an array of double values.
foreach (double retrySeconds in routingSettings.RetrySeconds)
{
if (retrySeconds >= 0) retryTimeSpans.Add(TimeSpan.FromSeconds(retrySeconds));
}
services.AddHttpClient<IRoutingClient, RoutingClient>()
.AddPolicyHandler((services, request) => HttpPolicyExtensions.HandleTransientHttpError()
.WaitAndRetryAsync(retryTimeSpans, onRetry: (outcome, timespan, retryAttempt, context) =>
{
services.GetService<ILogger<RoutingClient>>()?
.LogWarning($"Delaying for {timespan.TotalMilliseconds}ms, then making retry {retryAttempt}.");
}
));
return services;
}
I'd really like to use the Polly approach because it seems clean but I am not sure what I am doing wrong here. I must be doing something wrong because I would expect this to be a very common use-case for Polly that should be handled.
It turns out my problem was not caused by the code dealing with the response above. Instead, it was actually caused by my code that was manipulating the request.
I was reading the incoming request as an HttpRequestMessage object via the HttpContext.GetHttpRequestMessage() method and I attempted to re-use that same object to pass on to the backend service via a call to my typed HttpClient. However, the content stream for that request is read-once so I had to make a copy of that HttpRequestMessage as described in this answer to another post: https://stackoverflow.com/a/34049029/1221718
Here is a slightly more verbose version of that answer's code:
private static async Task<HttpRequestMessage> CloneHttpRequestMessageAsync(HttpRequestMessage request)
{
HttpRequestMessage copyOfRequest = new HttpRequestMessage(request.Method, request.RequestUri);
// Copy the request's content (via a MemoryStream) into the cloned object
var ms = new MemoryStream();
if (request.Content != null)
{
await request.Content.CopyToAsync(ms).ConfigureAwait(false);
ms.Position = 0;
copyOfRequest.Content = new StreamContent(ms);
// Copy the content headers
if (request.Content.Headers != null)
{
foreach (var h in request.Content.Headers)
{
copyOfRequest.Content.Headers.Add(h.Key, h.Value);
}
}
}
copyOfRequest.Version = request.Version;
foreach (KeyValuePair<string, object> prop in request.Properties)
{
copyOfRequest.Properties.Add(prop);
}
foreach (KeyValuePair<string, IEnumerable<string>> header in request.Headers)
{
copyOfRequest.Headers.TryAddWithoutValidation(header.Key, header.Value);
}
return copyOfRequest;
}