I have a pod with 3 containers.
Traffic comes into the Reverse Proxy, is validated, and then proxied onto the microservice. This is my most heavily used service and it starts to have this error after running for about a day.
System.Net.Http.HttpRequestException: Cannot assign requested address
---> System.Net.Sockets.SocketException (99): Cannot assign requested address
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
--- End of inner exception stack trace ---
at System.Net.Http.ConnectHelper.ConnectAsync(String host, Int32 port, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.ConnectAsync(HttpRequestMessage request, Boolean allowHttp2, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.CreateHttp11ConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.GetHttpConnectionAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpConnectionPool.SendWithRetryAsync(HttpRequestMessage request, Boolean doRequestAuth, CancellationToken cancellationToken)
at System.Net.Http.DiagnosticsHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
at System.Net.Http.HttpClient.FinishSendAsyncUnbuffered(Task`1 sendTask, HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
Restarting the pod is my only fix right now. I've goofed around looking at tcp statistics on the nodes, but it doesn't make sense that that would be the issue as killing the pod and restarting makes the problem go away.
I've also messed around with the httpclient available in .net core using the best practices with no change.
Any thoughts would be appreciated.
Upon further gathering of clues, I learned that these errors only show up while our REST microservice restarts (due to a memory leak). The error makes sense in context and I overestimated the severity of the issue.