The resident memory used by the java process, using G1GC, is much higher than the XMX and the memory reported by the NMT.
java -version
openjdk version "11.0.6" 2020-01-14 LTS
OpenJDK Runtime Environment 18.9 (build 11.0.6+10-LTS)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.6+10-LTS, mixed mode, sharing)
top output
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
787 root 20 0 99.9g 85.4g 22120 S 1053 66.8 11904:11 java
JVM Arguments
/usr/bin/java -Xms75g -Xmx75g -XX:NativeMemoryTracking=summary -XX:MaxGCPauseMillis=50
-XX:G1RSetSparseRegionEntries=4096 -XX:G1HeapRegionSize=32M
-Xlog:gc*=info,gc+phases=debug:file=/var/log/gc.log:time,uptimemillis,level,tags:filecount=5,filesize=50M -XX:+UseG1GC
Native Memory Tracking:
Total: reserved=81376MB, committed=80879MB
- Java Heap (reserved=76800MB, committed=76800MB)
(mmap: reserved=76800MB, committed=76800MB)
- Class (reserved=65MB, committed=65MB)
(classes #9993)
( instance classes #9216, array classes #777)
(malloc=1MB #29050)
(mmap: reserved=64MB, committed=63MB)
( Metadata: )
( reserved=64MB, committed=63MB)
( used=62MB)
( free=1MB)
( waste=0MB =0.00%)
- Thread (reserved=348MB, committed=35MB)
(thread #345)
(stack: reserved=346MB, committed=33MB)
(malloc=1MB #2072)
(arena=1MB #688)
- Code (reserved=244MB, committed=62MB)
(malloc=2MB #6831)
(mmap: reserved=242MB, committed=59MB)
- GC (reserved=3239MB, committed=3239MB)
(malloc=356MB #58172)
(mmap: reserved=2883MB, committed=2883MB)
- Compiler (reserved=2MB, committed=2MB)
(malloc=3MB #2963)
- Internal (reserved=16MB, committed=16MB)
(malloc=16MB #8415)
- Other (reserved=643MB, committed=643MB)
(malloc=643MB #296)
- Symbol (reserved=12MB, committed=12MB)
(malloc=10MB #122802)
(arena=3MB #1)
- Native Memory Tracking (reserved=4MB, committed=4MB)
(tracking overhead=4MB)
NMT reports around 80GB memory usage, however, the resident memory for the java process is over 85GB. Over time, the difference between the two keeps on increasing. For another VM running the same application, the resident memory went over 95GB while the NMT memory was again only 82GB. Due to this evergrowing memory footprint, eventually, the app gets killed by the OS oom-killer.
What could be contributing to the difference between the two? How it be reduced to have a more predictable memory footprint of the app?
"NMT in this release does not track third party native code memory allocations and JDK class libraries." https://docs.oracle.com/javase/8/docs/technotes/guides/vm/nmt-8.html
So basically third party native code you might access through the JNI, and things like ByteBuffers that explicitly allocate storage outside of the JVM.