Search code examples
javaeclipsememory-leaksjvmheap-dump

Java heap dump and the heap size after the heap analysis differs


I am experiencing memory leak and here goes some detail.

At the time of after-leak,

  • top shows 50GB memory as residential
  • heap dump file size is 25GB
  • eclipse MAT analyzer tells me the heap size is 10GB

At the time of before-leak,

  • top shows 30GB memory as residential
  • heap dump file size is 20GB
  • eclipse MAT analyzer tells me the heap size is 10GB

I am pretty surprised that the difference between top, heap-dump size, and the actual heap size. I am guessing that the difference between top and heap is the possibility of garbage collector heap and native heap areas. But, how come the heap dump file size and the actual heap size (from eclipse MAT analyzer) could differ?

Any insight on this problem?

UPDATE / ANSWER

Some of suggestions are to use jcmd (https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html) as the website tells "Native Memory Tracking". But, if you read the page carefully, you will see

Since NMT doesn't track memory allocations by non-JVM code, you may have to use tools supported by the operating system to detect memory leaks in native code.

So, in case of the leak inside the native library, jcmd is not an option.

After crawling the Internets for days and trying out various profilers, most effective for this problem is using jemalloc profiler.

This page helped me a lot! https://gdstechnology.blog.gov.uk/2015/12/11/using-jemalloc-to-get-to-the-bottom-of-a-memory-leak/


Solution

  • top and other OS level tools show how much system memory does your JVM process consume. Java heap, defined by -Xmx command line option, is only a part of that memory. Apart from heap JVM needs some memory for itself. Then there are java threads, each requiring a certain amount of memory. And Metaspace/Permanent Generation. And several others. You can read this blog post and this SO answer for more information.

    About the size of the dump file and the actual heap size the answer of @arnab-biswas is certainly true. MAT reports the size of actually used heap, consumed by live objects. But heap dump contains the whole of the heap, including garbage.