Search code examples
apache-sparkspark-submit

spark-submit config through file

I am trying to deploy spark job by using spark-submit which has bunch of parameters like

spark-submit --class Eventhub --master yarn --deploy-mode cluster --executor-memory 1024m --executor-cores 4 --files app.conf spark-hdfs-assembly-1.0.jar --conf "app.conf"

I was looking a way to put all these flags in file to pass to spark-submit to make my spark-submit command simple liek this

spark-submit --class Eventhub --master yarn --deploy-mode cluster --config-file my-app.cfg --files app.conf spark-hdfs-assembly-1.0.jar --conf "app.conf"

anyone know if this is possible ?

Solution

  • You can use --properties-file which should include parameters with starting keyword spark like 

    spark.driver.memory 5g
spark.executor.memory 10g

    And command should look like:

    spark-submit --class Eventhub --master yarn --deploy-mode cluster --properties-file <path-to-your-conf-file> --files app.conf spark-hdfs-assembly-1.0.jar --conf "app.conf"