I use expressjs for my server. A user hits a proxy service that hits mine. The proxy service is telling me that mine has given a 503. In this case, I can't really be logging that it happened since it seems express just error'ed out. All I know is that this other service receives a 503. Nowhere in my code do I set this up. I couldn't find anything when searching around, and if it weren't the proxy I wouldn't know the 503s are happening. In hundreds of thousands of requests, this happened just under 100 times.
I'm not expecting anyone to have the direct answer, however any clues would be much appreciated and any clue that gets to the answer will be marked as the answer with my comment below. Hoping some comments will help me give this question better information!
The solution relates to Envoy & Node. A coworker suggested server timeouts and I found https://github.com/envoyproxy/envoy/issues/1979.
In my case I added this: server.keepAliveTimeout = 0;
You can see docs over at https://nodejs.org/docs/latest-v10.x/api/http.html#http_server_keepalivetimeout.
I don't work on the infrastructure side of things so it took a bit for me to figure it was in that layer with folks on the teams who implemented the usage of Envoy.
When Envoy gets ECONNRESET it sends back a 503