Search code examples
javahadoopmapreduce

Reverse Sorting Reducer Keys

What is the best approach to get the Map Output keys to a reducer in reverse order? By default the reducer receives all keys in ascending order of keys. Any help or comments widely appreciated.

In simple words, in the normal scenario, if a map emits keys 1,4,3,5,2 the reducer receives the same as 1,2,3,4,5. I would like the reducer to receive 5,4,3,2,1 instead.

Solution

  • In Hadoop 1.X, you can specify a custom comparator class for your outputs using JobConf.setOutputKeyComparatorClass.

    Your comparator must implement the RawComparator interface.

    With Hadoop 2.X, this is done by using Job.setSortComparatorClass, still with an implementation of RawComparator.