I have a main thread which dispatches jobs to a thread pool. I'm using Java's Executor framework.
From the profiler (VirtualVM) I can see each thread's activity: I can see that the main thread is waiting a lot (because the executor's queue has a upper limit) which means the executor's queue is full most of the time. However the executor's threads are not as busy as I would have thought. Most of them have a waiting time of 75%. In virtualVM it says it waits on Monitor.
Can anyone explain why is this happenning? why would the executor threads wait while there is still plenty of work available to do? And how to improve the performance of the executor? thus to improve the performance overall? More detail on the executor's wait on monitor would be great.
The job runs in the workers is just some computation, which don't depends on anything else and don't communicate to any other thread (no synchronisation), except in the end, it put data in the database, using it is own connection.
After doing a thread dump, it turns out that it is the database layer that has the synchronisation. Hibernate's Sequence Generator is synchronised.
"pool-2-thread-1" - Thread t@13
java.lang.Thread.State: BLOCKED
at org.hibernate.id.SequenceHiLoGenerator.generate(SequenceHiLoGenerator.java:73)
- waiting to lock <61fcb35> (a org.hibernate.id.SequenceHiLoGenerator) owned by "pool-2-thread-5" t@23
at org.hibernate.internal.StatelessSessionImpl.insert(StatelessSessionImpl.java:117)
at org.hibernate.internal.StatelessSessionImpl.insert(StatelessSessionImpl.java:110)
at ac.uk.ebi.kraken.unisave.storage.impl.HibernateStorageEngine.saveEntryIndex(HibernateStorageEngine.java:269)
at ac.uk.ebi.kraken.unisave.storage.impl.EntryStoreImpl.storeEntryIndex(EntryStoreImpl.java:302)
at ac.uk.ebi.kraken.unisave.impl.MTEntryIndexLoader$EntryIndexLoader.run(MTEntryIndexLoader.java:129)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
Locked ownable synchronizers:
- locked <3d360c93> (a java.util.concurrent.ThreadPoolExecutor$Worker)