What is the proper value of HADOOP_HOME and PIG_CLASSPATH for APACHE HADOOP version 2.8.0?

I have the problem executing the Hadoop command from the PIG command line. The command and the error stack is below

My instructor suspects that it is because HADDOP_HOME and PIG_CLASSPATH are incorrect. I am on the HADOOP version 2.8.0.

So, originally I had HADOOP_HOME as

HADOOP_HOME=<CELLAR_DIRECTORY>/hadoop/2.8.0/

Then I switched the following setup:

HADOOP_HOME=<CELLAR_DIRECTORY>/hadoop/2.8.0/libexec/etc/hadoop

PIG_CLASSPATH is defined as $HADOOP_HOME

Commands I used in pig:

A = LOAD '/Users/anarinsky/Downloads/loaddata1.txt';

B = MAPREDUCE '/Users/anarinsky/workspace/wordcount/target/wordcount-1.jar' STORE A INTO '/Users/anarinsky/Downloads/tempwrite2' LOAD  '/Users/anarinsky/Downloads/tempwrite2' AS (word:chararray, count:int) `com.systemskills.hadoop.wordcount.WordCountDriver /wordcountdata /Users/anarinsky/Downloads/pigoptdir`;

Pig Stack Trace

ERROR 2025: Expected leaf of reduce plan to always be POStore. Found PONative

org.apache.pig.impl.logicalLayer.FrontendException: ERROR 1066: Unable to open iterator for alias B
    at org.apache.pig.PigServer.openIterator(PigServer.java:1019)
    at org.apache.pig.tools.grunt.GruntParser.processDump(GruntParser.java:747)
    at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:376)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:231)
    at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:206)
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:66)
    at org.apache.pig.Main.run(Main.java:564)
    at org.apache.pig.Main.main(Main.java:176)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:234)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:148)
Caused by: org.apache.pig.PigException: ERROR 1002: Unable to store alias B
    at org.apache.pig.PigServer.storeEx(PigServer.java:1122)
    at org.apache.pig.PigServer.store(PigServer.java:1081)
    at org.apache.pig.PigServer.openIterator(PigServer.java:994)
    ... 13 more
Caused by: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompilerException: ERROR 2025: Expected leaf of reduce plan to always be POStore. Found PONative
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler.compile(MRCompiler.java:321)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.compile(MapReduceLauncher.java:629)
    at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher.launchPig(MapReduceLauncher.java:152)
    at org.apache.pig.backend.hadoop.executionengine.HExecutionEngine.launchPig(HExecutionEngine.java:308)
    at org.apache.pig.PigServer.launchPlan(PigServer.java:1474)
    at org.apache.pig.PigServer.executeCompiledLogicalPlan(PigServer.java:1459)
    at org.apache.pig.PigServer.storeEx(PigServer.java:1118)
    ... 15 more

Solution

Alex! Unfortunately, it's not related to Pig paths (tried it on my configured hadoop cluster) with same result. The error you get refers to the fact that Physical plan compiler has a bug in compile method. So in order to make your attempt work you have two possibilities

Run native MR job using hadoop and after it finishes process it's results in pig

Edit pig source code and compile your own version. You'll need to edit org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler#compile method and replace

for (PhysicalOperator op : leaves) {
    if (!(op instanceof POStore)) {
        int errCode = 2025;
        String msg = "Expected leaf of reduce plan to " +
            "always be POStore. Found " + op.getClass().getSimpleName();
        throw new MRCompilerException(msg, errCode, PigException.BUG);
    }
}

with

    for (PhysicalOperator op : leaves) {
        if (!(op instanceof POStore) && !(op instanceof PONative)) {
            int errCode = 2025;
            String msg = "Expected leaf of reduce plan to " +
                "always be POStore. Found " + op.getClass().getSimpleName();
            throw new MRCompilerException(msg, errCode, PigException.BUG);
        }
    }