Search code examples
apache-sparkapache-spark-2.0spark-submit

Trouble Submitting Apache Spark Application to Containerized Cluster

I am having trouble running a Spark application using both spark-submit and the internal REST API. The deployment scenario I would like to demonstrate is Spark running as a cluster on my local laptop.

To that end, I've created two Docker containers for spark-master and spark-worker. Everything on that end seems to be working, the worker is able to register with master, both web consoles are available, and the internal REST (port 6066) API is responsive.

I've created an uber jar for my application, however, I get the same error in both cases when trying to execute it.

When using spark-submit, I use a local Spark installation on my local to submit to the container:

$SPARK_HOME/bin/spark-submit \
    --class com.reynoldsm88.jobs.SparkApp \
    --master spark://localhost:7077 \
    --deploy-mode cluster \
    --supervise \
    /Users/reynoldsm88/workspace/spark-app/spark-app/target/scala-2.11/spark-app-assembly-0.0.1-SNAPSHOT.jar

When using the REST API, my request looks like this:

curl -X POST http://localhost:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{
    "action" : "CreateSubmissionRequest",
    "appArgs" : [ "myAppArgument1" ],
    "appResource" : "/Users/reynoldsm88/workspace/spark-app/spark-app/target/scala-2.11/spark-app-assembly-0.0.1-SNAPSHOT.jar",
    "clientSparkVersion" : "2.2.0",
      "environmentVariables" : {
        "SPARK_ENV_LOADED" : "1"
    },
    "mainClass" : "com.reynoldsm88.jobs.SparkApp",
    "sparkProperties" : {
        "spark.master" : "spark://localhost:7077",
        "spark.jars" : "/Users/reynoldsm88/workspace/spark-app/spark-app/target/scala-2.11/spark-app-assembly-0.0.1-SNAPSHOT.jar",
        "spark.driver.supervise" : "false",
        "spark.app.name" : "MyJob",
        "spark.eventLog.enabled": "true",
        "spark.submit.deployMode" : "client"
    }
}'

The result for both is the same:

Spark Master (inside Docker container)

17/12/17 16:08:23 INFO Master: Driver submitted org.apache.spark.deploy.worker.DriverWrapper
17/12/17 16:08:23 INFO Master: Launching driver driver-20171217160823-0001 on worker worker-20171217155536-172.17.0.5-7078
17/12/17 16:08:24 INFO Master: Removing driver: driver-20171217160823-0001

Spark Worker (inside separate Docker container)

17/12/17 16:08:24 INFO Worker: Asked to launch driver driver-20171217160823-0001
17/12/17 16:08:24 INFO DriverRunner: Copying user jar file:///Users/reynoldsm88/workspace/spark-app/spark-app/target/scala-2.11/spark-app-assembly-0.0.1-SNAPSHOT.jar to /spark-2.2.0-bin-hadoop2.7/work/driver-20171217160823-0001/spark-app-assembly-0.0.1-SNAPSHOT.jar
17/12/17 16:08:24 INFO Utils: Copying /Users/reynoldsm88/workspace/spark-app/spark-app/target/scala-2.11/spark-app-assembly-0.0.1-SNAPSHOT.jar to /spark-2.2.0-bin-hadoop2.7/work/driver-20171217160823-0001/spark-app-assembly-0.0.1-SNAPSHOT.jar
17/12/17 16:08:24 INFO DriverRunner: Killing driver process!
17/12/17 16:08:24 WARN Worker: Driver driver-20171217160823-0001 failed with unrecoverable exception: java.nio.file.NoSuchFileException: /Users/reynoldsm88/workspace/spark-app/spark-app/target/scala-2.11/spark-app-assembly-0.0.1-SNAPSHOT.jar

I might be misunderstood, but I thought that in both of these cases the local JAR file would be submitted to Spark master in order to be distributed to the worker nodes. However, for some reason it looks as if the worker node is trying to load it from my local filesystem, which is problematic since the Docker container that Spark Worker is running is not aware of this file.

Perhaps I am mistaken as to how this works? If I truly wanted to Submit an application from my laptop to a containerized cluster do I need to mount that directory as a volume or host JAR somewhere so that the Worker can download it?

Any help or insight would be greatly appreciated.

Solution

  • Shortly after posting this I came up with a solution for my problem, but I am wondering if this is ideal or there is a better way. As part of the job script I copy the jar into a directory that is mounted as a host volume in Docker a la:

    Dockerfile

    ...
VOLUME /etc/spark/apps
...

    Start Script

    docker run -v /Users/reynoldsm88/spark/apps/:/etc/spark/apps --name spark-master -p 8080:8080 -p 7077:7077 -p 6066:6066 -t reynoldsm88/spark-master

    This seems to work and fits my needs at the moment. I do wish there was a way to deploy the JAR via an API to Spark, although it's possible I missed that in my research.