Jersey server CLOSE_WAIT leak after read timeout at the client

Question

Following is the (greatly simplified) code used for communicating from a client machine, to a REST server, using Jersey. There is a 5 minute timeout for establishing a connection at the start, and a 2 minute timeout for reading from the underlying socket.

public class JerseyClient {

  private final Map<Action, WebResource> resources;

  public JerseyClient(URI uri) {
    this.resources = new EnumMap<Action, WebResource>(Action.class);
    this.resources.put(Action.Put, getClient().resource(Action.Put.getURI()));
    this.resources.put(Action.Query, getClient().resource(Action.Query.getURI()));
  }

  private String submit(Action action, String input) throws Exception {
    WebResource resource = this.resources.get(action);
    ClientResponse response = null;
    synchronized (resource) {
      try {
          response = resource.accept(MediaType.APPLICATION_JSON_TYPE,
                  MediaType.TEXT_PLAIN_TYPE).type(MediaType.APPLICATION_JSON_TYPE).
                  post(ClientResponse.class, input);
          String responseString = null;
          // Handle the response and produce a response string...
        return responseString;
      } finally {
        if (response != null) {
          response.close();
        }
      }
    }
  }

  private static Client getClient() {
    Client client = Client.create();
    client.setReadTimeout(2*60*1000);
    client.setConnectTimeout(5*60*1000);
    return client;
  }

  private enum Action {
    Put, Query;
    public URI getURI(){
      switch (this) {
        case Put:
          return URI.create("PUT_URI");
        case Query:
          return URI.create("QUERY_URI");
        default:
          throw new InvalidStateException("Illegal action");
      }
    }
  }
}

The above code works as expected, unless the read timeout triggers at the client. In that case, a SocketTimeoutException is thrown and thus the response object in the submit() method of the JerseyClient class above remains null, so the underlying socket is never closed fully.

Apparently there is a partial closure of the socket by the client since, on the other side, the server enters the CLOSE_WAIT state (i.e., it has received a FIN packet from the client, as per the TCP specification). However, since it never gets the final ACK from the client (which should be sent if response.close() was called), it keeps the connection at CLOSE_WAIT (as shown by netstat), so each timed out REST call from the client potentially creates a dangling CLOSE_WAIT on the server.

Is there any way of solving the issue, without completely re-engineering the above code?

Answer 1

Your description doesn't make sense. The name of the state is CLOSE_WAIT, not CLOSED_WAIT, and it means what its (correct) name expresses: it is waiting for the local application to close the socket, after receiving a remote close from the peer.

If your server is entering CLOSE_WAIT:

1. The client has closed the socket.
2. The server has not closed the socket. This is a bug in the server.
3. The server will never get a final ACK from the client until it issues a FIN by closing the socket. The ACK is an acknowledgement of the FIN. Until the server issues the FIN there is nothing for the client to ACK.
4. It is the close that gets it out of CLOSE_WAIT, not the client ACK.
5. Calling response.close() at the client has nothing to do with the client sending the final ACK.