Service Fabric Asp.net Core Kestrel HttpClient hangs with minimal load

Question

I have a barebone Service Fabric Application hosting a Asp.net Core 1.1 Web API with Azure Application Gateway as reverse proxy on a Virtual Machine scale set of 5 DS3_V2.

The API have 10 HttpClients with different URLs injected via Dependency Injection. A simple foreach cycle in a method call 10 Httpclients in parallel:

        var cts = new CancellationTokenSource();
        cts.CancelAfter(600);

        //Logic for asyncronously parallel calling the Call method below

        public async Task<MyResponse> Call(CancellationTokenSource cts, HttpClient client, string endpoint )
        {
            var endpoint = "finalpartOfThendpoint";
            var jsonRequest = "jsonrequest";
            try
            {
                var content = new StringContent(jsonRequest, Encoding.UTF8, "application/json");
                await content.LoadIntoBufferAsync();
                if (cts.Token.IsCancellationRequested)
                {
                    return new MyResponse("Token Canceled");
                }
                var response = await client.PostAsync(endpoint, content, cts.Token);

                if (response.IsSuccessStatusCode && ((int)response.StatusCode != 204))
                {
                    //do something with response and return
                    return MyResponse("Response Ok")
                }
                return MyResponse("No response")
            }

            catch (OperationCanceledException e)
            {
                return new MyResponse("Timeout");
            }
        }

There is a single CancellationToken for all calls. After 600ms, the still pending HttpCalls are canceled and a response is sent back anyway. In local and in production all works perfectly, all endpoints are called and return in time, rarely one is canceled before the timeout.

But when the number of concurrent connections reach 30+, ALL calls timeout no matter what, until I reduce the load.

Does Asp.net Core have a connection limit?

This is how I create the HttpClients in a custom factory for injection in the main Controller:

 public static HttpClient CreateClient(string endpoint)
    {
     var client = new HttpClient
       {
          BaseAddress = new Uri(endpoint)
       };
       client.DefaultRequestHeaders.Accept.Clear();
       client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

        return client;
    }

All the Httpclients are reused and static. The same exact code works perfectly on a Asp.net Web API 2 hosted on OWIN in Service Fabric. The problem is only with Asp.net Core 1.1

I saw online to create a HttpClientHandler, but there is no parameter for concurrent connections.

What can I do to investigate further? No exception are thrown but the OperationcanceledException and If I remove the CancellationToken the calls are stuck and the CPU goes to 100%, basically 30 connections destroy the power of 5 quad core servers.

This has something to do to the number of calls going out of Kestrel.

---

UPDATE

I tried with WebListener and the problem is still present, so it's not Kestrel, but Asp.net Core

Answer 1

I figured it out.

Asp.net core still have some HttpClient limits for the connection to the same server like the old Asp.net WebAPI. It's poor documented but the old ServicepointManager option for maxconnections must now be passed via HttpClientHandler. I just create HttpClient like this and the problem vanished.

 var config = new HttpClientHandler()
            {
                MaxConnectionsPerServer = int.MaxValue
            };
            var client = new HttpClient(config)
            {
                BaseAddress = new Uri('url here')
            };

Really if someone of the team is reading, this should be the default.