I'm trying to diagnose a web service that sits behind some load balancers and proxies. Under load, one of the servers along the way starts to return HTTP 504 errors, which indicates a gateway timeout. With that background out of the way, here is my question:
When a proxy makes a request to the destination server, and the destination server receives the request but doesn't respond in time (thus exceeding the timeout), resulting in a 504, what happens when the destination server does eventually respond? Does it know somehow that the requestor is no longer interested in a response? Does it happily send a response with no idea that the gateway already sent HTTP error response back to the client? Any insight would be much appreciated.
It's implementation-dependent, but any proxy that conforms to RFC 2616 section 8.1.2.1 should include Connection: close on the 504 and close the connection back to the client so it can no longer be associated with anything coming back from the defunct server connection, which should also be closed. Under load there is the potential for race conditions in this scenario so you could be looking at a bug in your proxy.
If the client then wants to make further requests it'll create a new connection to the proxy which will result in a new connection to the backend.