Connection closed when sleeping long durations in between HttpClient requests

Question

I'm using Apache HttpComponents and specifically HttpClient for the first time. I have a fairly basic use case. I'm polling some third party server every N seconds (in this case their API is expecting a POST).

CloseableHttpClient httpclient = HttpClients.custom().
    setDefaultCookieStore(cookieStore).build();

Then I'm executing some POST requests in a loop... using:

while (!someCondition) {
    HttpPost httpPost = ...
    httpclient.execute(httpPost)
    Thread.sleep(SOME_TIME)
}

I noticed if I sleep for some longer duration, like 3 minutes, then I don't get a response back from the server and the connection dies consistently every time:

DEBUG [org.apache.http.wire] http-outgoing-1 << "end of stream"
DEBUG [org.apache.http.impl.conn.DefaultManagedHttpClientConnection] http-outgoing-1: Close connection
DEBUG [org.apache.http.impl.conn.DefaultManagedHttpClientConnection] http-outgoing-1: Shutdown connection
DEBUG [org.apache.http.impl.execchain.MainClientExec] Connection discarded
DEBUG [org.apache.http.impl.conn.DefaultManagedHttpClientConnection] http-outgoing-1: Close connection

I don't believe it's the server. I'm probably not using HttpComponents properly or I have it misconfigured. If I set it to a shorter duration like 1 minute, it works OK (I did notice it died after running ~15 minutes - so that's fifteen one minute intervals).

To send the request, I wrapped it in some Java 8 lambdas and utilize try with resources, but I don't think this matters:

private <R> R sendRequest(HttpUriRequest request, Function<String, R> func) {
    try {
        try (CloseableHttpResponse resp = this.httpclient.execute(request)) {
            HttpEntity entity = resp.getEntity();

            String responseString = EntityUtils.toString(entity);

            R result = null;
            if (func != null) {
                result = func.apply(responseString);
            }

            EntityUtils.consume(entity);
            return result;
        }
    } catch (IOException e) {
        e.printStackTrace();
        throw new RuntimeException("Error sending request: " + request, e);
    }
}

The actual exception is:

Caused by: org.apache.http.NoHttpResponseException: myserver.com:80 failed to respond
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:143)
    at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:57)
    at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:261)
    at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:165)
    at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:167)
    at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:272)
    at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:124)
    at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:271)
    at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:184)
    at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:88)
    at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
    at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)

Answer 1

It is very common and natural for HTTP servers to close out persistent connections that have been idle over a maximum period of inactivity in order to conserve resources. In your particular case if the client generates a request every 30 seconds the server keeps the connection alive, whereas if the connection stays idle longer it gets eventually closed on the server side (and becomes half-closed or 'stale' on the client side). Next time the client leases the connection from the pool and attempts to use it to execute a request, execution fails with an I/O exception. This is perfectly normal and can be expected. HttpClient tries to mitigate the problem by doing so-called stale connection check in order to find out whether or not the connection has been closed by the opposite endpoint or has become invalid otherwise. The 'stale' checks are relatively expensive and should be used sparingly. HttpClient as of version 4.4 seeks to improve performance out of the box by performing the check selectively. There was a defect in version 4.4 which basically disabled connection validation altogether. Please upgrade to version 4.4.1 and the problem should go away.

Having said all that I still think that the best course of action in your case is to evict all persistent connections from the connection pool prior to long periods of inactivity by using HttpClientConnectionManager#closeIdleConnections