SSLException, SocketException : Connection reset from Apache HttpClient

Question

We are using Apache HttpClient library to call HTTP REST endpoints in the Spring application. We are getting "Connection reset" error frequently now a days, We did not have any issue earlier but the error started appearing now, we see 1 to 5 out of 100 requests get this error. The code we are using to create HttpClient looks like this:

RequestConfig requestConfig = RequestConfig.custom()
                    .setConnectionRequestTimeout(connReqTimeoutInMillis)
                    .setConnectTimeout(connTimeoutInMillis)
                    .setSocketTimeout(socketTimeoutInMillis)
                    .build();

          CloseableHttpClient  client = HttpClientBuilder.create()
                    .setDefaultRequestConfig(requestConfig)
                    .setMaxConnPerRoute(numberOfConnections)
                    .setSSLSocketFactory(new SSLConnectionSocketFactory(SSLContext.getDefault(),
                            new String[]{"TLSv1.2"}, null,
                            ALLOW_ALL_HOSTNAME_VERIFIER)).build();

We are using this client object across the application. So the remote calls that we are making do not have any issues, as there are some calls we make from one service to another service within the project, those also have the same issue sometimes. This issue occurs for all HTTP method calls.

Exact error:

• javax.net.ssl.SSLException: java.net.SocketException: Connection reset
• Suppressed: java.net.SocketException: Broken pipe (Write failed)

Could someone please help here, Thanks for your help!

Answer 1

After doing a lot of analysis, finally, we could figure out the solution to this problem.

We have captured the TCP dump of all the connections made to the server and shared it with the server team. After analyzing the TCP dump we found that there were some connections in the client that were open more than the maximum "keep-alive" value at server side(in our case it was 500 seconds). and the connections that were open at our side were around 600 to 800 seconds, and when the same connection was used again the server was throwing a Connection reset.

The library we are using is Apache HttpClient and we discovered that if the server does not pass the "keep-alive" header in the response then the connection will remain open indefinitely on the client-side. And once this idle connection is used again, we face this issue.

So we have added a default keep-alive strategy in HTTP Client and also started a scheduler to clear idle connections beyond a threshold and finally we got rid of this issue. Both Solutions can be found here:

https://www.baeldung.com/httpclient-connection-management#keep-alive https://hc.apache.org/httpcomponents-client-4.5.x/current/tutorial/pdf/httpclient-tutorial.pdf