I am testing the use of Docker for executing Spark algorithms on MESOS. I managed to execute Spark in client mode with executors inside Docker, but I wanted to go further and have also my Driver running into a Docker Container. Here I ran into a behavior that I'm not sure is normal, let me try to explain.
I submit my spark application through MesosClusterDispatcher using a command like:
$ ./bin/spark-submit --class org.apache.spark.examples.SparkPi --master mesos://spark-master-1:7077 --deploy-mode cluster --conf spark.mesos.executor.docker.image=myuser/myimage:0.0.2 https://storage.googleapis.com/some-bucket/spark-examples-1.5.2-hadoop2.6.0.jar 10
My driver is running fine, inside its docker container, but the executors fail:
"sh: /some/spark/home/bin/spark-class: No such file or directory"
Looking on MESOS slaves log, I think that the executors do not run inside docker: "docker.cpp:775] No container info found, skipping launch". As my Mesos slaves do not have spark installed, it fails.
It seems that the spark conf that I gave in the first spark-submit is not transmitted to the Driver submitted conf, when launched in the docker container. The only workaround I found is to modify my Docker image in order to define inside its spark conf the spark.mesos.executor.docker.image property. This way, my executors get the conf well and are launched inside docker on Mesos. This seems a little complicated to me, and I feel the configuration passed to the early spark-submit should be transmitted to the Driver submit...
So my questions are:
https://issues.apache.org/jira/browse/SPARK-13258
:-)
I'll be fixing this soon. The workaround, as described in the JIRA, is to use SPARK_JAVA_OPTS rather than --conf.