Search code examples
apache-sparkhadoophadoop-yarn

YARN vs Spark processing engine based on real time application?


I understood YARN and Spark. But I want to know when I need to use Yarn and Spark processing engine. What are the different case studies in that I can identify the difference between YARN and Spark?


Solution

  • You cannot compare Yarn and Spark directly per se. Yarn is a distributed container manager, like Mesos for example, whereas Spark is a data processing tool. Spark can run on Yarn, the same way Hadoop Map Reduce can run on Yarn. It just happens that Hadoop Map Reduce is a feature that ships with Yarn, when Spark is not.

    If you mean comparing Map Reduce and Spark, I suggest reading this other answer.