Search code examples
apache-sparkcluster-computingmesosmesospheredcos

Submit job to DCOS Spark with multiple instances?

I have two instances of spark in my DCOS cluster, when I submit my job via CLI 

dcos spark run --submit-args="\
                 --driver-cores 8 \
                 --driver-memory 16384M \
                 --conf spark.eventLog.enabled=true \
                 --conf spark.eventLog.dir=hdfs://hdfs/history \
                 --class com.CalcPi \
                 <url to job -spark-test-assembly-0.0.5-SNAPSHOT.jar> 99000000"`

the job is forever stuck in the queue. But when I have only one instance everything works fine. I have already try --deploy-mode cluster --supervise

Solution

  • The following config options are hopefully the answer you are looking for:

    dcos config set spark.app_id spark-one
dcos spark run ...
dcos config set spark.app_id spark-two
dcos spark run ...