Increased latency by time - C# ASP.NET Core Web API

Question

I have an ASP.NET Core 3.1 Web API service hosted on Windows Server 2016 (IIS). It's a very simple web service that routes all incoming requests to a particular endpoint.

Here is the code:

using System;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Authentication;
using Microsoft.AspNetCore.Http;

namespace Proxy
{
    public class ProxyMiddleware
    {
        private readonly RequestDelegate next;
        private readonly string destinationPath;
        private readonly IProxyHttpClient client;

        public ProxyMiddleware(RequestDelegate next, IProxyHttpClient client)
        {
            this.next = next;
            this.client = client;
            this.destinationPath = "otherservice.svc";
        }

        public async Task Invoke(HttpContext context)
        {
            try
            {
                if (!context.Request.Path.ToString().ToLower().Contains(this.destinationPath))
                {
                    await next.Invoke(context);
                    return;
                }

                HttpResponseMessage response = await this.client.SendRequest(context.Request);
                context.Response.StatusCode = (int) response.StatusCode;

                await context.CopyProxyHttpResponse(response);
            }
            catch (HttpRequestException ex)
            {
                await context.Response.WriteAsync(ex.Message);
            }
            catch (Exception ex)
            {
                context.Response.StatusCode = (int) HttpStatusCode.InternalServerError;
                await context.Response.WriteAsync(ex.ToString());
            }
        }
    }
}

IProxyClient:

using System;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Http;

namespace Proxy
{
    public class ProxyHttpClient : IProxyHttpClient
    {
        private readonly string otherServiceUrl = "https://172.21.22.3:443";
        private readonly HttpClient httpClient;

        public ProxyHttpClient()
        {
            var handler = new HttpClientHandler
            {
                ClientCertificateOptions = ClientCertificateOption.Manual,
                ServerCertificateCustomValidationCallback = (httpRequestMessage, cert, cetChain, policyErrors) => true
            };
            this.httpClient = new HttpClient(handler);
        }

        public async Task<HttpResponseMessage> SendRequest(HttpRequest httpRequest)
        {
            HttpRequestMessage requestMessage = httpRequest.CreateProxyHttpRequest(new Uri(this.otherServiceUrl + httpRequest.Path + httpRequest.QueryString));

            return await this.httpClient.SendAsync(requestMessage);
        }
    }
}

Extensions:

using System;
using System.Net.Http;
using Microsoft.AspNetCore.Http;

namespace Proxy
{
    public static class HttpRequestExtensions
    {
        // obtained from https://github.com/aspnet/Proxy/blob/master/src/Microsoft.AspNetCore.Proxy/ProxyAdvancedExtensions.cs
        public static HttpRequestMessage CreateProxyHttpRequest(this HttpRequest request, Uri uri)
        {
            var requestMessage = new HttpRequestMessage();
            var requestMethod = request.Method;
            if (!HttpMethods.IsGet(requestMethod) &&
                !HttpMethods.IsHead(requestMethod) &&
                !HttpMethods.IsDelete(requestMethod) &&
                !HttpMethods.IsTrace(requestMethod))
            {
                var streamContent = new StreamContent(request.Body);
                requestMessage.Content = streamContent;
            }

            // Copy the request headers
            foreach (var header in request.Headers)
            {
                if (!requestMessage.Headers.TryAddWithoutValidation(header.Key, header.Value.ToArray()) && requestMessage.Content != null)
                {
                    requestMessage.Content?.Headers.TryAddWithoutValidation(header.Key, header.Value.ToArray());
                }
            }

            requestMessage.Headers.Host = uri.Authority;
            requestMessage.RequestUri = uri;
            requestMessage.Method = new HttpMethod(request.Method);

            return requestMessage;
        }

    }
}

.proj:

<Project Sdk="Microsoft.NET.Sdk.Web">

    <PropertyGroup>
        <TargetFramework>netcoreapp3.1</TargetFramework>
        <AspNetCoreHostingModel>OutOfProcess</AspNetCoreHostingModel>
        <AspNetCoreModuleName>AspNetCoreModuleV2</AspNetCoreModuleName>
    </PropertyGroup>

    <ItemGroup>
        <PackageReference Include="Microsoft.AspNetCore.Mvc.NewtonsoftJson" Version="3.1.22" />
        <PackageReference Include="Newtonsoft.Json" Version="13.0.1" />
        <PackageReference Include="Swashbuckle.AspNetCore" Version="5.6.3" />
    </ItemGroup>

</Project>

As said previously, I have this service deployed on IIS on windows server 2016. Once deployed, it works perfectly, keeps working well for 6-8 hours and as it keeps working, latency increases by time. It keeps increasing till it reaches the point of effectively being down. E.g. requests take 3+ minutes to get a reply (which effectively turns into a timeout). This behavior is consistent whether I call the proxy service from inside the server itself (localhost) or from outside the network via it's public IP.

Once I restart the server, it all goes back healthy! It stays healthy for 6-8 hours and the same behavior happens again. I have even tried restarting the service itself on IIS or recycling the application pool -> doesn't work. It must be the server (machine) itself.

We have checked the resource utilization on the server at the point of timeout increase, things look fine - around 50% utilization on memory (2GB out of 4GB) and CPU barely used.

Answer 1

From Yarp docs:

Middleware should avoid interacting with the request or response bodies. Bodies are not buffered by default, so interacting with them can prevent them from reaching their destinations. While enabling buffering is possible, it's discouraged as it can add significant memory and latency overhead. Using a wrapped, streaming approach is recommended if the body must be examined or modified. See the ResponseCompression middleware for an example.

I'd recommend to use this library instead of a custom solution. https://microsoft.github.io/reverse-proxy/articles/getting-started.html