Search code examples
apache-sparkhadoop-yarnspark-launcher

How to set driver java options in SparkLauncher

When using spark-submit to submit a Spark app to Yarn, I can pass java options to the driver via the --driver-java-options, for example: 

spark-submit --driver-java-options "-Dlog4j.configuration=file:///conf/log4j.properties" ...

How do I achieve the same when submitting via SparkLauncher? In particular, is there a way to achieve it with Spark 1.4?

Solution

  • Not familiar with SparkLauncher but from looking at the code it appears you can pass configuration with setConf(). In this if you add the property SparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS this should have the same effect.

    For example

    Process spark = new SparkLauncher()
     .setAppResource("/my/app.jar")
     .setMainClass("my.spark.app.Main")
     .setMaster("local[*]")
     .setConf(SparkLauncher.DRIVER_EXTRA_JAVA_OPTIONS, "-Dmy.property=someval")
     .launch();