Azure Function Proxy Internal Server Error 500 from SocketException

Question

I have an Azure Function that uses proxies and forwards to another azure function as the backend. There is an /api/ping endpoint that accepts a GET. When I send a HTTP-GET to ping I occasionally get a 500 Internal Server Error that it faulted where I only see the request on the proxy but I don't see the request on the backend code-executing function.

I added the header "Proxy-Trace-Enabled" for "true" to the header to trace the results. I have the results in my D:\home\LogFiles\Application\Proxies\DetailedTrace folder. In there the log for a failed request contains a "Backend" json object with the following

{
    "source": "forward-request",
    "timestamp": "2020-08-20T15:42:20.8272145Z",
    "elapsed": "00:00:00.0061051",
    "data": {
      "messages": [
        "Only one usage of each socket address (protocol/network address/port) is normally permitted Only one usage of each socket address (protocol/network address/port) is normally permitted",
        "Only one usage of each socket address (protocol/network address/port) is normally permitted",
        "Only one usage of each socket address (protocol/network address/port) is normally permitted"
      ]
    }
  }

I believe this is Azure Functions 1.0 on DotNet, but it was created a long time ago. Why is my simple Azure Function proxy giving me internal server errors that are not forwarding to my backend code to execute?

For reference on how to trace the requests

Answer 1

There are TCP Connection thresholds for Azure Function App Service Plans which correlate to the socket connections. The documentation was in a blog I'll link here. There was a similar question on TCP/Port Exhaustion that uses a similar correlation between issues. While the exception reported is different the errors go away in my testing when scaling up the app service it's on.

Example: I have 2 Azure Functions, FunctionA and FunctionB. FunctionA is a proxy and with no backend execution on App Service Plan P1. FunctionB is non-correlated function but executes on the same App Service Plan P1.

FunctionA under the App Service Plan P1 faults with Internal 500 Server Error issues when called. Reported as Faulted in App Insights, and traced in the backend logs as a socket exception.

I recreated the Azure Function, FunctionA, on App Service Plan P3. FunctionA did not receive 500 Internal Server Errors. However it does not require the scale of a P3 payment plan. So I moved it back to the P1. Server errors occurred again.

FunctionB was on the same App Service Plan as FunctionA (P1). Azure Monitoring Metrics summed up 4,200 SocketOutboundAll per minute. I moved(deleted and recreated) FunctionB from P1 to P3. I Kept FunctionA on P1. The errors of SocketOutboundAll have been eliminated from FunctionA on P1. Functions on P3 App Service Plan are not reporting any exceptions either.