Search code examples
hadoopmapreducedistributed-computing

In hadoop, I just want to execute my own custom program on each node


Yes, I want to run my custom program on each hadoop node. I want to deploy No mapper and reducer. It is just like distributed computing system that doesn't works like mapreduce(but which uses hdfs internally). How should I do?


Solution

  • Both MapReduce and Tez jobs use YARN (Yet Another Resource Negotiator) to get distributed and executed over the cluster in so-called containers. You can also use YARN yourself to run your own jobs instead. Please take a look at the Hadoop Architecture Overview for a high-level overview.