I am using VisualVM to monitor my JBoss instance. I have attached a screenshot of it aswell.
The problem is after I restart the JBoss instance, the CPU on the OS starts to go high. Load can go as high as 40 and JAVA process in top command shows upto 300% usage.
This then goes on to slow down the application at the front end.
VisualVM shows that CPU is high and that thread count is increasing also.
How can I further go to the root cause of this ?
When it comes to high CPU usage, I generally look for two things :
To diagnose #1 further, enabling GC Logging may be the best way. However, you can use jstat as well:
jstat -gc PID 60s
The above command will collect the memory usage and garbage collection details from the JVM every 60s. In a debugging environment, 60s should be OK, but in a production environment 3600s should be sufficient. jstat being very light weight, doesn't create any adverse impact. The output will help you to understand if there are frequent GCs (major/minor). Frequent major collection is a problem for sure (it pauses the application), however, very frequent minor collection could also cause high CPU (application is creating too much of garbage too frequently). If this is the case, probably you need a head dump and you need to understand application's memory usage details, but not before that. Please remember, capturing heap dump may "hang" your application (I don't suggest it in production unless you can restart the application immediately after capturing the data).
To diagnose #2, "top" provides an option ("H") to check the CPU consumed by individual threads. That will point (in real time) the application threads which are consuming CPU. Also (as other's have suggested), capture 5/6 thread dumps in an interval of 10 seconds each. Looks for threads in RUNNABLE state. These are the threads which are doing work and hence occupying CPU. Is the same (set of) thread stuck in the Runnable state over multiple dumps? Probably you have got a problem.
Hope this helps. Best of luck!
