Search code examples
hadoopmapreducebigdataavromrunit

How to test reducer with avro params in MRUnit?

I'm trying to test the Reducer bellow

Reducer<CompositeKeyWritable, AvroValue<Class1>, AvroKey<Class2>, NullWritable>

For these purpose I use the next code in the test:

CustomReducer reducer = new CustomReducer();
    reduceDriver = new ReduceDriver<CompositeKeyWritable, AvroValue<Class1>, AvroKey<Class2>, NullWritable>();

    Job job = new Job();

    reduceDriver = ReduceDriver.newReduceDriver(reducer).withConfiguration(job.getConfiguration());
    reduceDriver.setReducer(reducer);

    job.setInputFormatClass(AvroKeyInputFormat.class);

    job.setMapOutputKeyClass(CompositeKeyWritable.class);
    AvroJob.setMapOutputValueSchema(job, union);

    AvroJob.setOutputKeySchema(job, Class2.SCHEMA$);
    job.setOutputFormatClass(AvroKeyOutputFormat.class);
    job.setOutputValueClass(NullWritable.class);

As a result i got the next error:

java.lang.RuntimeException: java.io.EOFException
at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:86)
at org.apache.hadoop.mrunit.internal.io.Serialization.copy(Serialization.java:97)
at org.apache.hadoop.mrunit.internal.io.Serialization.copyWithConf(Serialization.java:110)
at org.apache.hadoop.mrunit.TestDriver.copy(TestDriver.java:675)
at org.apache.hadoop.mrunit.ReduceDriverBase.addInput(ReduceDriverBase.java:167)
at org.apache.hadoop.mrunit.ReduceDriverBase.addInput(ReduceDriverBase.java:181)
at com.wellcare.caregap.hadoop.reducers.CareGapReducerTest.testReducer(CareGapReducerTest.java:70)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
at java.lang.reflect.Method.invoke(Unknown Source)
at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:50)
at org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:459)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:675)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:382)
at org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:192)

What is wrong in configuration of job? I will be very thankfull if some body help me with these Exception.

Solution

  • Debug your method readFields and write in Class1. It seems it is trying to read or write fields after reaching the end of the DataInput or DataOutput stream in one of them when you are adding your input with reduceDriver.withInput(yourkey, yourvalue);