I am using RestTemplate to talk to APIs. Recently i encountered this issue, where almost 90% of the threads are in waiting state.
p685934-2328" #2328 prio=5 os_prio=0 tid=0x00000000023b6000 nid=0x2f1f waiting on condition [0x00007f6df703f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000000ab0eaae8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at org.apache.http.pool.AbstractConnPool.getPoolEntryBlocking(AbstractConnPool.java:380)
at org.apache.http.pool.AbstractConnPool.access$200(AbstractConnPool.java:69)
at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:246)
- locked <0x00000000e728c1e8> (a org.apache.http.pool.AbstractConnPool$2)
at org.apache.http.pool.AbstractConnPool$2.get(AbstractConnPool.java:193)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:303)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionManager.java:279)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:191)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87)
at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53)
at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:723)
at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:680)
at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:600)
RestTemplate is defined like below.
HttpComponentsClientHttpRequestFactory clientHttpRequestFactory = new HttpComponentsClientHttpRequestFactory();
clientHttpRequestFactory.setConnectTimeout( 3000 );
clientHttpRequestFactory.setReadTimeout( 3000 );
SSLContext sslContext = SSLContexts.custom()
.loadTrustMaterial((X509Certificate[] chain, String authType) -> true)
.build();
SSLConnectionSocketFactory sslsf = new SSLConnectionSocketFactory(
sslContext, (String hostname, SSLSession session) -> true);
clientHttpRequestFactory.setHttpClient(HttpClients.custom()
.setSSLSocketFactory(sslsf).build());
RestTemplate restTemplate = new RestTemplate(clientHttpRequestFactory);
Spring boot version : 2.0.3.RELEASE
<dependency>
<groupId>org.apache.httpcomponents</groupId>
<artifactId>httpclient</artifactId>
<version>4.5.5</version>
</dependency>
More than 500 requests per minute will go from RestTemplate.
Any help on what went wrong with this API. will RestTemplate do not close the socket in any scenario?
Thanks, Hari
I think the reason 90% of the threads are in a locking state is that the threads are blocked / waiting for the API to respond. The RestTemplate is blocking or synchronous per the documentation:
You could switch to the asynchronous or non-blocking WebClient and you should get much better throughput per thread:
Another option would be to spawn a thread per connection if that is possible but I think the asynchronous client will scale better if you have the ability to port your code.
Cheers,
Kyle