Search code examples
apache-sparkapache-zeppelin

SPARK_SUBMIT_OPTIONS didn't work in Apache Zeppelin 0.7.3

Trying to run Spark Zeppelin on Yarn cluster, I set the following variables in zeppeline-env.sh:

export HADOOP_CONF_DIR=/opt/hadoop/etc/hadoop
export SPARK_HOME=/opt/spark
export SPARK_SUBMIT_OPTIONS="--executor-memory 512m --driver-memory 512m --executor-cores 2 --queue onlineQ"

But it didn't use the SPARK_SUBMIT_OPTIONS values in deploying job on YARN! For example executor-memory is 1024m as default. When I set executor-memory in the Zeppelin interpreter setting, it will be applied. As I should set --queue property for Spark Submit, is there any other way (than SPARK_SUBMIT_OPTIONS variable) to set it?

Solution

  • You can set in spark interpreter settings.

    spark.driver.memory etc.

    More information here: https://zeppelin.apache.org/docs/0.7.2/interpreter/spark.html#configuration