My server (.war) running on jboss sometimes comes to 100% cpu. I made a dump with jstack, but don't succeed to see an indication to what is taking 100% cpu. jstack -F don't succeed, so i runned it with no flag. There more line but i can't post all of it...
I have an idea but not sure this is right: Could it be because of all the lock on 0x00002aaab58c0900?
There are a few more runnable like the following one
"Connect thread 120.18.1.17 session" prio=10 tid=0x00002aaad41ce800 nid=0x604b runnable [0x00000000448d8000..0x00000000448d8b10]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at com.jcraft.jsch.IO.getByte(IO.java:82)
at com.jcraft.jsch.Session.read(Session.java:821)
at com.jcraft.jsch.Session.run(Session.java:1241)
at java.lang.Thread.run(Thread.java:619)
"Thread-28" prio=10 tid=0x00002aaad4e9e400 nid=0x6409 runnable [0x0000000044cdc000..0x0000000044cdcd10]
java.lang.Thread.State: RUNNABLE
at com.company.departement.application.infrastruct.SSHAgent.executeCommand(SSHAgent.java:128)
at com.company.departement.application.server.DataCollector.getcomponentKPIs(DataCollector.java:3345)
- locked <0x00002aaab58c0900> (a com.company.departement.application.server.DataCollector)
at com.company.departement.application.server.DataCollector.getAllDataRun(DataCollector.java:2021)
- locked <0x00002aaab58c0900> (a com.company.departement.application.server.DataCollector)
- locked <0x00002aaab5e6e768> (a java.lang.Object)
at com.company.departement.application.server.Initapplication$DataThread.run(Initapplication.java:12)
Here is the beggining of the dump
2012-05-02 15:02:25
Full thread dump Java HotSpot(TM) 64-Bit Server VM (11.0-b16 mixed mode):
"Attach Listener" daemon prio=10 tid=0x000000005fa00400 nid=0x6f17 waiting on condition [0x0000000000000000..0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"http-0.0.0.0-8180-13" daemon prio=10 tid=0x00002aaadc2c1400 nid=0x277c waiting for monitor entry [0x00000000460ef000..0x00000000460f0a90]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.company.departement.application.server.DataCollector.componentKPIsToClient(DataCollector.java:1413)
- waiting to lock <0x00002aaab58c0900> (a com.company.departement.application.server.DataCollector)
at com.company.departement.application.server.getComponentKPIsServiceImpl.getComponentKPIs(getComponentKPIsServiceImpl.java:20)
at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:569)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:153)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
at java.lang.Thread.run(Thread.java:619)
"http-0.0.0.0-8180-12" daemon prio=10 tid=0x00002aaadc2bf800 nid=0x2663 waiting for monitor entry [0x0000000045fee000..0x0000000045fefa10]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.company.departement.application.server.DataCollector.componentKPIsToClient(DataCollector.java:1413)
- waiting to lock <0x00002aaab58c0900> (a com.company.departement.application.server.DataCollector)
at com.company.departement.application.server.getComponentKPIsServiceImpl.getComponentKPIs(getComponentKPIsServiceImpl.java:20)
at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:569)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:153)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
at java.lang.Thread.run(Thread.java:619)
"http-0.0.0.0-8180-11" daemon prio=10 tid=0x00002aaadc036000 nid=0x25b7 in Object.wait() [0x0000000045eee000..0x0000000045eeeb90]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab67e1c98> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
- locked <0x00002aaab67e1c98> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
at java.lang.Thread.run(Thread.java:619)
"http-0.0.0.0-8180-10" daemon prio=10 tid=0x00002aaadc034800 nid=0x24a8 waiting for monitor entry [0x0000000045dec000..0x0000000045dedb10]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.company.departement.application.server.DataCollector.componentKPIsToClient(DataCollector.java:1413)
- waiting to lock <0x00002aaab58c0900> (a com.company.departement.application.server.DataCollector)
at com.company.departement.application.server.getComponentKPIsServiceImpl.getComponentKPIs(getComponentKPIsServiceImpl.java:20)
at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:569)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:153)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
at java.lang.Thread.run(Thread.java:619)
"http-0.0.0.0-8180-9" daemon prio=10 tid=0x00002aaadc034000 nid=0x244d in Object.wait() [0x0000000045cec000..0x0000000045cecc90]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab67e3c98> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
- locked <0x00002aaab67e3c98> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
at java.lang.Thread.run(Thread.java:619)
"http-0.0.0.0-8180-8" daemon prio=10 tid=0x00002aaadc02e800 nid=0x233e in Object.wait() [0x00000000458e8000..0x00000000458e8c10]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab67e5c98> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
- locked <0x00002aaab67e5c98> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
at java.lang.Thread.run(Thread.java:619)
"http-0.0.0.0-8180-7" daemon prio=10 tid=0x00002aaadc002800 nid=0x6edd in Object.wait() [0x0000000045beb000..0x0000000045bebc90]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab5e57af8> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
- locked <0x00002aaab5e57af8> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
at java.lang.Thread.run(Thread.java:619)
"http-0.0.0.0-8180-6" daemon prio=10 tid=0x00002aaadc000c00 nid=0x6edc in Object.wait() [0x0000000045aea000..0x0000000045aeac10]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab5e6b0e8> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
- locked <0x00002aaab5e6b0e8> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
at java.lang.Thread.run(Thread.java:619)
"http-0.0.0.0-8180-5" daemon prio=10 tid=0x00002aaadc010000 nid=0x6edb waiting for monitor entry [0x00000000459e8000..0x00000000459e9d90]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.company.departement.application.server.DataCollector.componentKPIsToClient(DataCollector.java:1413)
- waiting to lock <0x00002aaab58c0900> (a com.company.departement.application.server.DataCollector)
at com.company.departement.application.server.getComponentKPIsServiceImpl.getComponentKPIs(getComponentKPIsServiceImpl.java:20)
at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:569)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:153)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
at java.lang.Thread.run(Thread.java:619)
"http-0.0.0.0-8180-4" daemon prio=10 tid=0x00002aaadc00ec00 nid=0x6da0 waiting for monitor entry [0x00000000457e6000..0x00000000457e7a90]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.company.departement.application.server.DataCollector.componentKPIsToClient(DataCollector.java:1413)
- waiting to lock <0x00002aaab58c0900> (a com.company.departement.application.server.DataCollector)
at com.company.departement.application.server.getComponentKPIsServiceImpl.getComponentKPIs(getComponentKPIsServiceImpl.java:20)
at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:569)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:153)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
at java.lang.Thread.run(Thread.java:619)
"http-0.0.0.0-8180-3" daemon prio=10 tid=0x00002aaadc026800 nid=0x6d9f waiting for monitor entry [0x0000000041506000..0x0000000041507a10]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.company.departement.application.server.DataCollector.componentKPIsToClient(DataCollector.java:1413)
- waiting to lock <0x00002aaab58c0900> (a com.company.departement.application.server.DataCollector)
at com.company.departement.application.server.getComponentKPIsServiceImpl.getComponentKPIs(getComponentKPIsServiceImpl.java:20)
at sun.reflect.GeneratedMethodAccessor78.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.google.gwt.user.server.rpc.RPC.invokeAndEncodeResponse(RPC.java:569)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processCall(RemoteServiceServlet.java:208)
at com.google.gwt.user.server.rpc.RemoteServiceServlet.processPost(RemoteServiceServlet.java:248)
at com.google.gwt.user.server.rpc.AbstractRemoteServiceServlet.doPost(AbstractRemoteServiceServlet.java:62)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:810)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:252)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.jboss.web.tomcat.filters.ReplyHeaderFilter.doFilter(ReplyHeaderFilter.java:81)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
at org.jboss.web.tomcat.security.CustomPrincipalValve.invoke(CustomPrincipalValve.java:39)
at org.jboss.web.tomcat.security.SecurityAssociationValve.invoke(SecurityAssociationValve.java:153)
at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
at java.lang.Thread.run(Thread.java:619)
"http-0.0.0.0-8180-2" daemon prio=10 tid=0x00002aaadc026000 nid=0x6d9c in Object.wait() [0x0000000044bdb000..0x0000000044bdbb90]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab5ef7ac0> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
- locked <0x00002aaab5ef7ac0> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
at java.lang.Thread.run(Thread.java:619)
"TP-Monitor" daemon prio=10 tid=0x00002aaad4d84c00 nid=0x6414 in Object.wait() [0x00000000456e6000..0x00000000456e6c90]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab5e594c0> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable.run(ThreadPool.java:559)
- locked <0x00002aaab5e594c0> (a org.apache.tomcat.util.threads.ThreadPool$MonitorRunnable)
at java.lang.Thread.run(Thread.java:619)
"TP-Processor4" daemon prio=10 tid=0x00002aaad4ffb400 nid=0x6413 runnable [0x00000000455e5000..0x00000000455e5c10]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)
- locked <0x00002aaab5e6eb10> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:453)
at java.net.ServerSocket.accept(ServerSocket.java:421)
at org.apache.jk.common.ChannelSocket.accept(ChannelSocket.java:295)
at org.apache.jk.common.ChannelSocket.acceptConnections(ChannelSocket.java:641)
at org.apache.jk.common.SocketAcceptor.runIt(ChannelSocket.java:852)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
at java.lang.Thread.run(Thread.java:619)
"TP-Processor3" daemon prio=10 tid=0x00002aaad4909800 nid=0x6412 in Object.wait() [0x00000000454e4000..0x00000000454e4d90]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab5e58698> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x00002aaab5e58698> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:619)
"TP-Processor2" daemon prio=10 tid=0x00002aaad5147800 nid=0x6411 in Object.wait() [0x00000000453e3000..0x00000000453e3d10]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab5e58b50> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x00002aaab5e58b50> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:619)
"TP-Processor1" daemon prio=10 tid=0x00002aaad5013c00 nid=0x6410 in Object.wait() [0x00000000452e2000..0x00000000452e2a90]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab5e59008> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:656)
- locked <0x00002aaab5e59008> (a org.apache.tomcat.util.threads.ThreadPool$ControlRunnable)
at java.lang.Thread.run(Thread.java:619)
"http-0.0.0.0-8180-1" daemon prio=10 tid=0x00002aaadc025400 nid=0x640f in Object.wait() [0x00000000451e1000..0x00000000451e1a10]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00002aaab5f20ee8> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
at java.lang.Object.wait(Object.java:485)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.await(MasterSlaveWorkerThread.java:81)
- locked <0x00002aaab5f20ee8> (a org.apache.tomcat.util.net.MasterSlaveWorkerThread)
at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:107)
at java.lang.Thread.run(Thread.java:619)
There are 2 distinct problems here.
You have a stuck or long running thread which is blocking other requests from completing. The long running task (DataCollector.getAllDataRun()) is blocking the other requests from completing. You saw this in your initial assessment. This is most likely the cause for your client not getting a response because these new requests are running into the lock (see all the Waiting to Lock)that the getAllDataRun() created.
There could be a few reasons for this thread to be blocking.
The 2nd part is the CPU at 100%. The Waiting to Lock threads are probably not using any CPU cycles as they wait to get the signal that they are free to continue processing. The most likely culprit here is the SSHAgent.executeMethod() method. You will want to see what the method's doing along with the DataCollector.getAllDataRun() method.
If you are running this on a unix based OS, you can read another answer I posted here.
prstat on Solaris or top [shift+h when it opens] on other unix based OS's will give you a break down of the CPU activity right down to the thread id within the JVM which is giving you the issue. You can pair this information with your thread dump to determine exactly which piece of code is causing your problem.