Search code examples
apache-sparkapache-spark-standalone

What is the difference between --master local[n] and --total-executor-core = n (Spark Standalone)?

I have a Spark Standalone cluster with 4 nodes, each has 56 cores

when I run my same job with --master local[56] and master --spark://... --executor-cores 56 --total-executor-cores 56 (which I think are the same)

I find their performances are different, while the later one performs better. What is the difference between these 2 kinds of spark-submit?

Solution

  • What is the difference between these 2 kinds of spark-submit?

    --master local[56] uses a single JVM with 56 threads for executors.

    --master spark://... uses a Spark Standalone cluster that may or may not be running on the same machine where you execute spark-submit. That cluster may or may not have multiple distributed nodes each with one or more CPUs.

    --total-executor-cores NUM is for the total cores for all executors. It does not say how many executors (nodes) you get for a single Spark application. It may be one, but could also be tens or more (each with unknown number of CPUs). What you know is the total number of CPU cores for this Spark application.

    --executor-cores NUM is for the number of cores per executor. (default: all available cores on the worker in standalone mode).

    The last option begs for an explanation of the differences between executors and workers.

    Executors are JVM processes hosted by workers. Executors are responsible for executing tasks which represent your Spark application. Workers are part of Spark Standalone cluster.

    You may have 10 executors and 1 worker or 1 executor and 10 workers. That is also possible to have 10 CPU cores for your Spark application out of 20 CPU cores available in the entire Spark Standalone cluster. This is to let other Spark applications to be spark-submited to the cluster to share workload.

    One could ask:

    So according to this seems as if --total-executor-cores is equal to --executor-cores, then we could just have 1 executor, thus only one node would be activated for the Spark jobs

    --total-executor-cores is total number of CPU cores per Spark application while --executor-cores is the number of CPU cores per one single executor of a Spark application.

    In other words, --total-executor-cores is usually larger than --executor-cores, but they both could also be the same (e.g. with one single worker and one single executor).