Search code examples
javagarbage-collectionjava-17zgc

GC warning about "memory mappings per process" and JVM crash after migrating to java 17

We are migrating our java 8 application to java 17, and switching the GC from G1GC to ZGC. Our application is running as a container and the only difference between base-images of these two is the version of java. For example for java 17 version:

FROM ubuntu:20.04
...
ENV JAVA_HOME /usr/lib/jvm/jdk-17.0.5
ENV PATH $JAVA_HOME/bin:$PATH

And its related java options:

-Xmx100g
-XX:+UseZGC

When running application, these gc warnings appear:

[0.018s][warning][gc] ***** WARNING! INCORRECT SYSTEM CONFIGURATION DETECTED! *****
[0.018s][warning][gc] The system limit on number of memory mappings per process might be too low for the given
[0.018s][warning][gc] max Java heap size (102400M). Please adjust /proc/sys/vm/max_map_count to allow for at
[0.018s][warning][gc] least 184320 mappings (current limit is 65530). Continuing execution with the current
[0.018s][warning][gc] limit could lead to a premature OutOfMemoryError being thrown, due to failure to map memory.

And after a few hours, after memory reached about ~60GB, application crashed:

Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(...) failed; error='Not enough space' (errno=12)
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(...) failed; error='Not enough space' (errno=12)
# fatal error: Failed to map memory (Not enough space)
# A fatal error has been detected by the Java Runtime Environment:
[error occurred during error reporting (), id 0xb, SIGSEGV (0xb) at pc=0x00007fc929746941]

But our java 8 version of application had never encountered such a problem before. I know adjusting vm.max_map_count will most likely fix the problem', but my question is what exactly has been changed in java or GC which causes these warnings and eventually these errors?

Solution

  • but my question is what exactly has been changed in java or GC which causes these warnings and eventually these errors?

    It's simply an implementation detail of the ZGC garbage collector. It needs more fine-grained control over its memory mappings which means a larger total amount of mappings.

    The older collectors usually reserved huge contiguous spaces of address space and only rarely punched some holes into those ranges to release memory back to the OS. ZGC works with non-contiguous ranges of varying size which means it has to request more individual mappings from the OS.