Search code examples
mapreducehbasehadoop2

How to detect last record in mapper while fetching data from HBase Table

I am trying to read records from HBase table in my java mapper. I want to detect the last record to perform an operation on that. How can we identify the last record. I have a zero reducer. I cannot hold all the records in memory.

Solution

  • You can determine when a mapper has processed its final record when it calls its cleanup method: 

    cleanup(org.apache.hadoop.mapreduce.Mapper.Context context)

    https://hadoop.apache.org/docs/r2.6.2/api/org/apache/hadoop/mapreduce/Mapper.html

    This will give you the ability to determine per-mapper when it has processed all of its key/values and take some action.

    If you're trying to aggregate without using a reduce stage and you're reading from HBase i would assume you have some guarantees about the groupings of the data that would also help. For example you could look for a change of key entering a mapper and you would know you have received all keys that are the same.