Search code examples
apache-sparkhadoopmapreducehadoop-yarnhadoop2

ApplicationMaster - Is it different for different types of applications which run in YARN?

I am trying to understand YARN and came to know of the term ApplicationMaster along with other terms.

As I understand in Hadoop 1.x there was only MR which could be run, however as needs grow, YARN was created to make it possible to run different types of jobs, for example, MR Job, Spark based jobs etc.

My question is:

Does each type of jobs (MR, Spark etc) have their own ApplicationMaster, that is, for MR is there one kind of ApplicationMaster, and for Spark based job, another type of ApplicationMaster?

Solution

  • Answer to your question is Yes.
    Want to write a new Application on Yarn? Hadoop documentation has a section explaining how.
    This article from Cloudera is helpful in understanding the YARN application launch process.

    Want to have a peek into Spark's implementation of the Yarn ApplicationMaster. Check here