I am trying to use counters within my MapReduce program, but whenever I am trying to increment it I am getting following error:
14/04/18 12:22:51 INFO mapred.JobClient: Task Id : attempt_201404172237_0052_m_000003_0, Status : FAILED
Error: org.apache.hadoop.mapreduce.Counter
And then when I'm trying to read the counter's value I'm getting following exception:
Exception in thread "main" java.lang.IncompatibleClassChangeError: org.apache.hadoop.mapreduce.Counter
at com.zikesjan.bigdata.TfIdfMain.main(TfIdfMain.java:72)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:60)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
at java.lang.reflect.Method.invoke(Method.java:611)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
It is happening when I am trying to implement it from both Mapper or Reducer. My implementation looks as follows:
In case when I use counter from Mapper it's just one line in the map method that looks like this:
context.getCounter(MyCounters.Documents).increment(1);
In case when I have tried it out from Reducer it was in cleanup:
public void cleanup(Context context){
context.getCounter(MyCounters.Documents).increment(numberOfRows);
}
And then I have implemented the counter's Enum like this:
public enum MyCounters {
Documents
}
And in my main Class I would like to retrieve counter's value as follows:
long documents = countLines.getCounters().findCounter(MyCounters.Documents).getValue();
Unfortunately it seems like there aren't working for me any operations with the counters. Is there some other specific way how the counters must be initialized except that I have described above?
I am using Hadoop version 1.1.1 on IBM BigInsights instance (if this information is anyhow relevant to the problem). Particularly when I type hadoop version I got:
Hadoop 1.1.1
Subversion git://dasani.svl.ibm.com/ on branch (no branch) -r f0025c9fd25730e3c1bfebceeeeb50d930b4fbaa
Compiled by jenkins on Fri Aug 9 17:06:14 PDT 2013
From source with checksum 21fb4557d5057d18b673b3fd46176f95
Thank you in advance for any help.
EDIT: I have tried my Map Reduce program on my toy one node Cloudera Hadoop instance that I have in the virtual box and it seems like there it works as I expected. After the hadoop version command here I got:
Hadoop 2.0.0-cdh4.4.0
Subversion file:///data/1/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hadoop-2.0.0-cdh4.4.0/src/hadoop-common-project/hadoop-common -r c0eba6cd38c984557e96a16ccd7356b7de835e79
Compiled by jenkins on Tue Sep 3 19:33:17 PDT 2013
From source with checksum ac7e170aa709b3ace13dc5f775487180
So my questions are:
1) Is the reason why counters do work for me at Cloudera only that it is single node instance? Or are counters supposed to work on multi node instances? And thus there is only problem on the side of the IBM BigInsights?
No, the issue has nothing to with single node instance. You need to upgrade your Hadoop version that is running on IBM Biginsights. It is succeeding on Cloudera's Sandbox because it is running Hadoop 2.
Hadoop 2 API is incompatible with Hadoop 1 API.