adding external property file to classpath in spark

Question

I am currently submitting my fat jar to spark cluster using below command. Application fat jar and related configuration are in the folder /home/myapplication

$SPARK_HOME/bin/spark-submit --jars $SPARK_HOME/lib/protobuf-java-2.5.0.jar --class MainClass /home/myapplication/my-application-fat.jar -appconf /home/myapplication/application-prop.properties -conf /home/myapplication/application-configuration.conf

Now my requirement is to add an external property file /home/myapplication/external-prop.properties to classpath of both driver and worker node.

I searched lot of resources but could not get right solution i am looking for!

Please help in resolving the issue. Thanks in advance

Answer 1

your requirement lies in using spark.executor.extraClassPath configuration to point to the properties file. But before that as @philantrovert has pointed out to use --files option to copy the property file to the worker nodes.

So your correct command should be

$SPARK_HOME/bin/spark-submit --jars $SPARK_HOME/lib/protobuf-java-2.5.0.jar --class MainClass /home/myapplication/my-application-fat.jar -appconf /home/myapplication/application-prop.properties -conf /home/myapplication/application-configuration.conf --files /home/myapplication/external-prop.properties --conf "spark.executor.extraClassPath=./"