We have WebSphere environment with 2 Node Agents and 4 Application Servers. On high traffic one of the application servers stops responding to requests at it jumps to maximum web container thread counts.
On analyzing thread dump we found that approx 60% of threads are in runnable state with 20% each of Wait and Parked state.
We do not see any deadlock warning in thread dump.
On closely look we have found that one of the web container thread owns the lock with below message:
Owns Monitor Lock on com/ibm/ws/classloader/ExtJarClassLoader@0x0A00000000FA6F30
Could someone help with the understanding of the above error and its resolution?
It is important to look at the stack trace of thread that owns the lock, and then look at the stack trace of all the other threads waiting on that lock. For ExtJarClassLoader, it almost certainly means that all the threads are attempting to perform loadClass operations. If many threads are attempting this and getting blocked, then this typically means the running code is attempting a failing loadClass operation, catching the ClassNotFoundException, and continuing. Creating and throwing ClassNotFoundException is expensive, so the code should typically be modified to cache the overall result (e.g., if it is trying a sequence of class loads, the positive/negative outcome should somehow be cached). This will of course be complicated if the code calling loadClass is a third party library.