What is use of Yarn/any queue while executing spark-submit?

Question

I am trying to understand why do we need to give a queue while executing spark-submit? What role does it play and why is needed and how is it used? I am trying to understand spark and relatively new to it. So please point me if there is an answer/explanation already

PS: Couldn't find any proper explanation for the above question and thus posting in this community

Answer 1

If you're talking about the queue parameter in YARN mode then it is

The name of the YARN queue to which the application is submitted. To get more details about the subject you can look at:

• https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html
• https://docs.cloudera.com/HDPDocuments/HDP2/HDP-2.6.5/bk_yarn-resource-management/content/setting_up_queues.html
• https://blog.cloudera.com/yarn-capacity-scheduler/

But in general, the YARN queue is just a way to isolate resource usage between different processes.