Search code examples
herokujvmjmxjconsole

Heroku default JVM options duplicated at beginning and end of VM options

My spring boot application is running against the dreadful R14 memory error. Locally, I can run my app fine with -Xmx150m, so I wanted to override Heroku's default -Xmx300m configuration, using JAVA_TOOL_OPTIONS.

In the Heroku tool options window, I have set JAVA_TOOL_OPTIONS to: -Xmx150m -XX:MaxMetaspaceSize=100m

To track memory usage in the Heroku UI, I have also enabled https://devcenter.heroku.com/articles/language-runtime-metrics-jvm, which I understand, also changes the VM arguments.

To be able to use jconsole, I enabled Heroku CLI Java plugin: https://devcenter.heroku.com/articles/exec#using-java-debugging-tools

Still I'm hitting the R14 errors, and it looks like my -Xmx150m definition doesn't take effect.

Using jconsole, I could see that the VM arguments actually in use, are the following:

VM arguments:  -XX:+UseContainerSupport -Xmx300m -Xss512k -XX:CICompilerCount=2 -Dfile.encoding=UTF-8 -javaagent:/app/.heroku/bin/heroku-metrics-agent.jar -Xmx150m -XX:MaxMetaspaceSize=100m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1098 -Dcom.sun.management.jmxremote.rmi.port=1099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=true -Djava.rmi.server.hostname=172.17.39.70 -Djava.rmi.server.port=1099 -Dserver.port=36924 -XX:+UseContainerSupport -Xmx300m -Xss512k -XX:CICompilerCount=2 -Dfile.encoding=UTF-8

Let's break this down:

  1. Heroku default VM options for Java: -XX:+UseContainerSupport -Xmx300m -Xss512k -XX:CICompilerCount=2 -Dfile.encoding=UTF-8
  2. Heroku options for runtime metrics: -javaagent:/app/.heroku/bin/heroku-metrics-agent.jar
  3. My own custom Java Tool Options: -Xmx150m -XX:MaxMetaspaceSize=100m
  4. Heroku options for CLI Java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1098 -Dcom.sun.management.jmxremote.rmi.port=1099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=true -Djava.rmi.server.hostname=172.17.39.70 -Djava.rmi.server.port=1099 -Dserver.port=36924
  5. AGAIN, Heroku default VM options ??? -XX:+UseContainerSupport -Xmx300m -Xss512k -XX:CICompilerCount=2 -Dfile.encoding=UTF-8

Now I tried to disable Heroku's runtime metrics, and this shows the following VM arguments in jconsole:

XX:+UseContainerSupport -Xmx300m -Xss512k -XX:CICompilerCount=2 -Dfile.encoding=UTF-8 -Xmx150m -XX:MaxMetaspaceSize=100m -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1098 -Dcom.sun.management.jmxremote.rmi.port=1099 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=true -Djava.rmi.server.hostname=172.18.116.38 -Djava.rmi.server.port=1099 -Dserver.port=35883 -XX:+UseContainerSupport -Xmx300m -Xss512k -XX:CICompilerCount=2 -Dfile.encoding=UTF-8

===> this only disabled number 2 from the list above.

Which finally brings me to my question: how can you avoid Heroku adding it's own default options, both at the start and end of the option list???

Solution

  • After opening a support ticket with Heroku about this, I found that following approach works:

    1. Remove any custom settings from JAVA_TOOL_OPTIONS
    2. Configure all desired jvm params, including the heroku default ones, in JAVA_OPTS.

    I now have java opts set to -XX:+UseContainerSupport -Xmx150m -Xss512k -XX:CICompilerCount=2 -Dfile.encoding=UTF-8 and this works like a charm.

    I have configured this through https://dashboard.heroku.com/apps/<<name of your app>>/settings => Config Vars