Search code examples
javamultithreadingmongodbdeadlock

Deadlock in mongodb java driver

I have multithreaded java application with my ~5 threads (and also many threads from jetty web server), some of them are reading/writing mongodb from time to time. Some of writes are intensive, where I read 200K mongodb objects, but they don't happen continiously, they happen once in few minutes. For few hours application works perfectly, but later I see this situation:

enter image description here

Mongo is not doing any work, as far I understand it:

enter image description here

Here is my jstack output:

https://gist.github.com/stiv-yakovenko/06b0d235fd2c32d839788edf56aaa6cd

You can see that all threads are waiting for one thread, which, in turn is waiting for mongo, while mongo is doing nothing. Before problem begings, healthy situation is that no threads are waiting for anyone else, because load is not that high to block everything. Before mongo I was using mapdb to store same data and I never had issues like that.

I've seen same situation with multiple threads waiting for mongo, so I decided to put all mongodb invocations under the same ReentrantLock(true). I hoped that rootcause was too many threads wanted to access mongo, but it doesn't help. I don't know what to do, tried to reproduce the problem with simple code, but I can't. Any ideas?

UPD: here is jstat output as one of commenters requested:

enter image description here

Solution

  • Well, finally it turned out that it was a garbage collection. I've ended up using G1 garbage collector. But it was not enough, because it couldn't deliver required latency (though it was close to it). I had to split application into two parts, one for doing intensive garbage-producing calculations, another for low-latency web responses.