Our application consists (database-wise) of three parts. A legacy-part using EJB, the current application using hibernate/spring and a third-party-library using Mybatis. Database operations in one session are coordinated by JTA. The application is equipped with a rudimentary APM.
We have recently seen a very strange behavior but were not yet able to reproduce and analyze it: Our application usually performs ok hibernate-wise but in that case there was some load from other sessions on all parts of the application. During this event the APM reported times for executing transactions in the EJB/Spring part multiplied tenfold. We saw heavy CPU usage for those threads executing the EJB/Spring transactions even when the actual code executed within the transaction was minimal.
After analysis we learned that those monitored times include our business code as well as the time spent for Hibernate-Flush. We also saw a correlation between the amount of elements within the entitymanager and the time spent for completing the transaction.
Our current suspicion is, that for some reason flush is running berserk.
Has anyone seen something like this or has any idea what might be the reason?
The answer was completely elsewhere. We have stumbled over the JDK7/CodeCache Bug, which results in the JIT Compiler completely stopping to compile anything, once CodeCache has filled up. This results in a lot of code being executed in interpretation mode and therefore quite a bit slower than expected.
That made the flushing slow and appear as the evil-doer.
If you are seeing strange slow-downs of your app and are still running JDK7 take a look at your CodeCache memory area. If it has filled up to 100% and then suddenly dropped to ~50-66& and no longer growing anymore, you hit that bug.
Hint: A more lengthy explanation can be found here: JDK7 Application is getting slow after some Uptime