Running Ignite inside kubernetes environment, but after a while when the load is increased getting below Blocked Thread in Thread dump,
qtp102185114-154-acceptor-2@6275fad4-ServerConnector@6a3aa688{HTTP/1.1}{0.0.0.0:8080} - priority:5 - threadId:0x00007f78a6e1c000 - nativeId:0x84 - state:BLOCKED
stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:234)
- waiting to lock <0x00000000a049f430> (a java.lang.Object)
at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
Can anyone explain what's going wrong here? Given Blocked thread is on every node of a cluster.
It looks like Jetty (which is used for REST API internally in Ignite) doesn't have enough threads in the pool to accept all incoming connection when loading increases. I would suggest checking CPU utilization firstly and it is not 100% then you are able to increase threads count in Jetty XML configuration