Search code examples
javaout-of-memorygoogle-cloud-run

Google Cloud Run Out of Memory Issue

I'm trying to understand why I'm getting out of memory errors in my cloud run application when cloud run never shows my memory allocation > 36% (purple line belo).

java.lang.OutOfMemoryError: Java heap space

enter image description here

Teal line is CPU usage and the purple is allocated memory. This scenario seems like we're CPU bound, but during this time period we are logging the java.lang.OutOfMemoryError errors. I'm pretty baffled. When exceeding max heap I'm used to seeing CPU usage rise during garbage collection and then a steep drop off in memory usage once garbage collection completes. The graph seems like we do not have a memory consumption issue because the memory allocation is staying pretty flat at 36%.

How can be we be blowing up the heap size if we are only reporting 36% of available memory allocated?

Solution

  • This turns out to be me not understanding how JVMs default their max heap size based on the memory available to container they are running in. By default JVMs will set a max heap size of 25% of the memory available to the container. If your container has 1GB memory available the JVM will set a max heap size of 250Mb.

    I solved this by adding the following line to my docker file:

    ENV JAVA_TOOL_OPTIONS "-XX:MaxRAMPercentage=75 -XX:+ExitOnOutOfMemoryError"

    Which tells the JVM to set the max heap size of 75 percent of the memory available to the container. No more OOM errors.