Search code examples
javahadoopmapreduceclouderahapi

Add the hapi 3rd-party lib (jars) to map-reduce

I am trying to add the hapi jars to the map-reduce task. I have all the required jars under /home/training/lib
I make the fat jar which contain all the hapi jars that i need and run
hadoop jar mp.jar stubs.MsgParser infile outfile -libjars /home/training/lib i got this Usage: MsgParser [generic options] <input dir> <output dir> and in my driver i implemented the ToolRunner
More Info:

  • I am running on CloudEra distribution.
  • i need the jars only in the mapper
  • mp.jar is exported jar from Eclipse

Is my hadoop jar command need modification or is there any other solution to add these jars, i can't copy the jars to /usr/lib/hadoop/client-0.20 cause i am not the owner
Thank you in advance!

Solution

  • Solution: upload your jars to hdfs and simply add 

    DistributedCache.addFileToClassPath(new Path(HDFSPATH), conf);

    in the main method of the driver