.NET HttpClient slow or not responding in Azure WebApp

Question

We are running a WebJob and Api on Azure App Services. Some of the WebJobs perform REST Calls to third party services, like ebay. All worked fine, until a few days ago, when the services started throwing this error randomly:

{\"ClassName\":\"System.Net.Http.HttpRequestException\",\"Message\":\"An error occurred while sending the request.\",\"Data\":{},\"InnerException\":{\"ClassName\":\"System.Net.WebException\",\"Message\":\"The underlying connection was closed: An unexpected error occurred on a receive.\",\"Data\":{},\"InnerException\":{\"ClassName\":\"System.IO.IOException\",\"Message\":\"Unable to read data from the transport connection: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.\",\"Data\":{},\"InnerException\":{\"NativeErrorCode\":10060,\"ClassName\":\"System.Net.Sockets.SocketException\",\"Message\":\"A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond\",\"Data\":{},\"InnerException\":null,\"HelpURL\":null,\"StackTraceString\":\"   at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)\\r\\n   at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)\",\"RemoteStackTraceString\":null,\"RemoteStackIndex\":0,\"ExceptionMethod\":\"8\\nEndReceive\\nSystem, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\\nSystem.Net.Sockets.Socket\\nInt32 EndReceive(System.IAsyncResult)\",\"HResult\":-2147467259,\"Source\":\"System\",\"WatsonBuckets\":null},\"HelpURL\":null,\"StackTraceString\":\"   at System.Net.Security._SslStream.EndRead(IAsyncResult asyncResult)\\r\\n   at System.Net.TlsStream.EndRead(IAsyncResult asyncResult)\\r\\n   at System.Net.Connection.ReadCallback(IAsyncResult asyncResult)\",\"RemoteStackTraceString\":null,\"RemoteStackIndex\":0,\"ExceptionMethod\":\"8\\nEndRead\\nSystem, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089\\nSystem.Net.Security._SslStream\\nInt32 EndRead(System.IAsyncResult)\",\"HResult\":-2146232800,\"Source\":\"System\",\"WatsonBuckets\":null}

The calls sometimes work, but are very slow and sometimes return the error. Running a local instance of the service results in no failure. Only in the production environment, we have these issues.

We use a singleton instance of the HttpClient to perform the calls.

public sealed class Client : HttpClient
    {
        private static volatile Client _instance = new Client();

        static Client()
        {
        }

        private Client() : base(new NativeMessageHandler())
        {
            // limit the connections in parallel to 100 by default
            ServicePointManager.ServerCertificateValidationCallback = delegate { return true; };
            ServicePointManager.SecurityProtocol = SecurityProtocolType.Ssl3 | SecurityProtocolType.Tls | SecurityProtocolType.Tls11 | SecurityProtocolType.Tls12;

            // if this setting does not work, follow these instructions on app.config
            // ServicePointManager.DefaultConnectionLimit needs to be set before the ServicePoint is created 
            ServicePointManager.DefaultConnectionLimit = 100;
        }

        public static Client Instance => _instance;
    }

We call the endpoint using the Client like this:

var client = Client.Instance;
var authenticationHeader = new AuthenticationHeaderValue("Bearer", token.AuthToken);
var url = "https://api.ebay.com/sell/account/v1/fulfillment_policy?marketplace_id=EBAY_DE";
var response = await client.GetMessageAsync(url, m => m.Headers.Authorization = authenticationHeader);

The GetMessageAsync Method is an Extension Method and just performs the action to set the header.

The problems started shortly after Microsoft announced this Security patch: https://learn.microsoft.com/answers/questions/6842/announcement-samesite-cookie-handling-and-net-fram.html

The Client is set to accept TLS 1.2 and 1.1.

Answer 1

In terms to narrow down possible causes, I had ebay and Microsoft Support to check their systems. It turned out to actually be an issue on side of ebay.