karate-gatling: how to resolve java heap space OutOfMemoryError?

Question

Currently I'm trying to run our functional tests (about 300 requests) with 10 users in parallel using gatling-plugin

mvn clean test-compile gatling:test -Dkarate.env=test

with the following .mvn/jvm.config local maven options in the project folder:

-d64 -Xmx4g -Xms1g -XshowSettings:vm -Djava.awt.headless=true

At some point while processing some big response in parallel the gatling process is aborted:

[ERROR] Failed to execute goal io.gatling:gatling-maven-plugin:3.0.2:test (default-cli) on project np.rest-testing: Gatling failed.: Process exited with an error: -1 (Exit value: -1) -> [Help 1]

with the following stack trace:

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid25960.hprof ...
Heap dump file created [1611661680 bytes in 18.184 secs]
Uncaught error from thread [GatlingSystem-scheduler-1]: Java heap space, shutting down JVM since 'akka.jvm-exit-on-fatal-error' is enabled for ActorSystem[GatlingSystem]
java.lang.OutOfMemoryError: Java heap space
        at akka.actor.LightArrayRevolverScheduler$$anon$3.nextTick(LightArrayRevolverScheduler.scala:269)
        at akka.actor.LightArrayRevolverScheduler$$anon$3.run(LightArrayRevolverScheduler.scala:235)
        at java.lang.Thread.run(Thread.java:748)

I have tried to increase heap space to 10 GB (-Xmx10g) in different ways:

1. via environment property MAVEN_OPTS=-Xmx10g
2. via local project maven options .mvn/jvm.config
3. via maven-surefire-plugin configuration as suggested here

Although 10GB is allocated for maven process as you can see at the start of maven process:

VM settings:
    Min. Heap Size: 1.00G
    Max. Heap Size: 10.00G
    Ergonomics Machine Class: client
    Using VM: Java HotSpot(TM) 64-Bit Server VM

but the OutOfMemoryError is still thrown during each gatling-plugin execution.

When analyzing each heap dump eclipse memory analyzer indicates always the same results:

84 instances of "com.intuit.karate.core.StepResult", loaded by "sun.misc.Launcher$AppClassLoader @ 0xc0000000" occupy 954 286 864 (90,44 %) bytes. 

Biggest instances:
•com.intuit.karate.core.StepResult @ 0xfb93ced8 - 87 239 976 (8,27 %) bytes... 

What can be done to reduce the heap space usage and prevent OutOfMemoryError? Can someone share some thoughts and experience?

Answer 1

After some investigations I've finally noticed, that heap dump shows always 1GB. That means the increased heap space is not used by gatling-plugin.

By adding the following jvm argument to the plugin, the problem is solved even with 4GB:

<jvmArgs>
    <jvmArg>-Xmx4g</jvmArg>
</jvmArgs>

So, with the following gatling-plugin configuration the error doesn't appear any more:

         <plugin>
            <groupId>io.gatling</groupId>
            <artifactId>gatling-maven-plugin</artifactId>
            <version>${gatling.plugin.version}</version>
            <configuration>
                <simulationsFolder>src/test/java</simulationsFolder>
                <includes>
                    <include>performance.test.workflow.WorkflowSimulation</include>
                </includes>
                <compilerJvmArgs>
                    <compilerJvmArg>-Xmx512m</compilerJvmArg>
                </compilerJvmArgs>
                <jvmArgs>
                    <jvmArg>-Xmx4g</jvmArg>
                </jvmArgs>
            </configuration>
        </plugin>