Search code examples
apache-sparkpysparkgarbage-collectiong1gc

Using G1GC garbage collector with spark 2.3

I am trying to use the G1GC garbage collector for spark job but I get a

Error: Invalid argument to --conf: -XX:+UseG1GC

I tried using these options but haven't been able to get it working

spark-submit --master spark://192.168.60.20:7077 --conf -XX:+UseG1GC /appdata/bblite-codebase/test.py

and 

spark-submit --master spark://192.168.60.20:7077 -XX:+UseG1GC /appdata/bblite-codebase/test.py

What is the correct way to call a G1GC collector from spark?

Solution

  • JVM options should be passed as spark.executor.extraJavaOptions / spark.driver.extraJavaOptions, ie.

     --conf "spark.executor.extraJavaOptions=-XX:+UseG1GC"