I have allocated a max value of 2048m for my Java web application. Due to some memory leak, the application has consumed nearly 2 GB of allocated memory. At this point, I have taken a memory dump using jvisualvm. During one instance the heap dump size was near > 1.5 GB whereas in the other instance the heap dump size was < 100 MB. What could be the reason behind this?
Heap is not the only memory JVM process consumes. There are memory consumed by stacks, compiled code and class meta data, GC data and native memory that is used by native libraries that your application uses. To explain each one further;
Stacks; each thread in your application have its own stack and stack size can be configured with -Xss switch. If your application has 20 threads and stack size is 1 mb; 20 mb will be used by stacks.
Compiled code; native code that is generated by JIT compiler. (from byte-codes)
Class meta data; with java 8, class meta data is stored in the native part of JVM.
GC data; GC needs space to keep track of objects to be collected. ( like card tables)
Native libraries(ex: gdal, ffmpeg ..) that your application uses, allocate memory on the native heap, which will increase the memory that is consumed by the JVM process.
You can analyze internal JVM native memory with NMT JVM feature. https://docs.oracle.com/javase/8/docs/technotes/guides/troubleshoot/tooldescr007.html
Also check these articles if you need more info.
https://plumbr.io/blog/memory-leaks/why-does-my-java-process-consume-more-memory-than-xmx
http://hirt.se/blog/?p=401
https://dzone.com/articles/java-8-permgen-metaspace