I am running a medium sized jersey web app under tomcat. I find out that the app keeps crashing after certain amount of time of execution (couple of days) due to out of memory problem, I already increased the heap size but that is not the problem as I am facing a memory leak somewhere.
I looked for ways to debug this to no avail. I m using a tool called YouKit Java to help me into this and I realised that the used heap memory keeps growing indefinitly until it breaks. The garbage collection doesnt seem to run at any moment.
heap memory usage afer 16h I have run a debug app for the entire night and even with minimal to no use this happens: used heap grows from a couple of MB to XXXMB (> 1GB in prod) with no load on it at all. After forcing garbage collection, memory usage goes back to normal.
The next picture shows the used memory growing again after my forced GC with basic usage of the app: page reload, some get queries that returns data from db (sqlite) and some post query that write into db and open some sockets. Note that for everything that I tested, I also run the opposite command that should cancel my changes. but the memory just keeps increasing.
I did a memory snapshot to browse through what is instanciated. Biggest object- dominators shows huge arborescence of object java.lang.ref.Finalizer, knowing that I dont ever call any finalize method (not that I know of at least)
So I am very lost in this, Java is not my biggest strength and I am having lot of hard times debugging this. I am wondering if there is something possibily preventing GC from running and causing this ? (As I see that after force execution of GC things fall back to a more normal state) Can this be caused by TOMCAT or jersey itself ?
Side notes about the app : It is an API that lets you create tcp tunnel in background (server and client sockets). Every tunnel is spawned in a thread. It also does some data fetching and writing to an sqlite db. I tried to be sure that everything is closed properly (db connections, queries, sockets and is unreference...) when the work is done. For the tunnels, I am relying on a lightly edited version of a library called javatunnel (it may be also bew the culprit but couldn't find anything proving it)
The socket are perhaps not correctly closed so that the garbage collector cannot freed them and the related objects ? I think you should try to investigate deeper in this way.
You can activate garbage collector logs to detect when it runs. You have to add flags -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCTimeStamps at the JVM start-up.
