We are having a performance Issue in our project which seems to originate (at least in parts) from the way Hibernate uses the classloader. This was discovered in Java thread dumps which were taken during high load tests on our internal environment. The JVM dumped is the JVM of the Weblogic managed server which runs the application, and dumps were taken while the monitoring dashboard shows hogging threads and pending user requests.
Example:
"[ACTIVE] ExecuteThread: '126' for queue: 'weblogic.kernel.Default (self-tuning)'" daemon prio=10 tid=0x00007f2fe9486000 nid=0x663b waiting for monitor entry [0x00007f2faeae6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ClassLoader.loadClass(ClassLoader.java:405)
- waiting to lock <0x000000078c0d76b0> (a weblogic.utils.classloaders.GenericClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at weblogic.utils.classloaders.GenericClassLoader.loadClass(GenericClassLoader.java:178)
at org.hibernate.internal.util.ReflectHelper.classForName(ReflectHelper.java:187)
at org.hibernate.internal.util.ReflectHelper.getConstantValue(ReflectHelper.java:278)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl$JavaConstantConverter.handleDotStructure(QueryTranslatorImpl.java:592)
at org.hibernate.hql.internal.ast.QueryTranslatorImpl$JavaConstantConverter.visit(QueryTranslatorImpl.java:587)
What we can see in these thread dumps (using Samurai/TDA) is that there seem to be a great many threads at any time which are waiting for locks on the classloader. This classloader, provided by WLS, seems to be synchronized - which explains the locking/blocking thread pattern...
It seems Hibernate uses the classloader to evaluate expressions in queries. So I'm not sure the classloader calls actually load any new classes.
The problem is the number of calls to the classloader which appear to be going on all the time... At times I observed as much as 30% of the total number of threads (~30-40+ our of 130) waiting on getting a classloader lock!
--> It seems that the synchronization of the WLS classloader leads to a very high internal overhead when a high number of threads are trying to service a high user load (i.e. many Hibernate queries).
Right now this synchronized classloader issue seems to be the main reason which limits the throughput of our application, leading to degrading performance under heavy load. Also the problem does not go away if we scale up CPU/memory or various WLS specific pools (like EJB/JDBC connections/...) -- as it is specific to the entire JVM we run our app in.
I'd much appreciate your input on the topic.
P.S.
Google seemed to show that we are not the first ones to encounter this issue (e.g. this mailing list question or this Oracle Support Question), but there is no real solution/explanation to this problem.
The problem is that application programmers think that Class.forName() and Classloader.loadClass() are cheap operations like a new Object() would be. And application servers thinks that they are a rare startup operation.
For hibernate using criteria or dynamic generated JPQL can trigger this lock contention http://dimovelev.blogspot.dk/2015/02/performance-pitfalls-hibernate-criteria.html