I have a weblogic clustered environment with 2 instances of servers. One of which is getting hanged and other one is working perfectly fine. After analysing the thread dumps I can see that almost 90% of the threads are BLOCKED and after tracing the threads it comes to below stack trace.
"[ACTIVE] ExecuteThread: '330' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f9494376000 nid=0x66a2 runnable [0x00007f944bb78000]
java.lang.Thread.State: RUNNABLE
at java.io.UnixFileSystem.getBooleanAttributes0(Native Method)
at java.io.UnixFileSystem.getBooleanAttributes(UnixFileSystem.java:242)
at java.io.File.exists(File.java:813)
at sun.misc.URLClassPath$FileLoader.getResource(URLClassPath.java:1080)
at sun.misc.URLClassPath.getResource(URLClassPath.java:199)
at java.net.URLClassLoader$1.run(URLClassLoader.java:358)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
Please find below the thread dump for the BLOCKED thread.
"[ACTIVE] ExecuteThread: '298' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f9494352000 nid=0x6276 waiting for monitor entry [0x00007f944e09e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
- waiting to lock <0x0000000719e7ded8> (a weblogic.utils.classloaders.GenericClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:179)
at ch.qos.logback.classic.spi.PackagingDataCalculator.loadClass(PackagingDataCalculator.java:207)
at ch.qos.logback.classic.spi.PackagingDataCalculator.bestEffortLoadClass(PackagingDataCalculator.java:232)
at ch.qos.logback.classic.spi.PackagingDataCalculator.computeBySTEP(PackagingDataCalculator.java:138)
at ch.qos.logback.classic.spi.PackagingDataCalculator.populateUncommonFrames(PackagingDataCalculator.java:113)
at ch.qos.logback.classic.spi.PackagingDataCalculator.populateFrames(PackagingDataCalculator.java:105)
at ch.qos.logback.classic.spi.PackagingDataCalculator.calculate(PackagingDataCalculator.java:57)
at ch.qos.logback.classic.spi.ThrowableProxy.calculatePackagingData(ThrowableProxy.java:147)
at ch.qos.logback.classic.spi.LoggingEvent. (LoggingEvent.java:124)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:440)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:396)
at ch.qos.logback.classic.Logger.warn(Logger.java:713)
This stack trace is similar for all other BLOCKED threads as well.
Please let me know if anyone has come across such issue. Any help is much appreciated.
The thread dumps indicate that your threads are blocked while trying to write log messages. Logback needs some class and asks a class loader to load it. Somewhere down the class loaders hierarchy, one of the class loaders is stuck while loading a library and it causes all other threads to hang waiting for this loader.
Most likely, you have a dependency library that is placed on a mounted network disk that is either unreachable or extremely slow or has some bad blocks. It causes the hang on reading this library. Check your classpath, check disks and the network.