HPCC/HDFS Connector

Does anyone know about HPCC/HDFS connector.we are using both HPCC and HADOOP.There is one utility(HPCC/HDFS connector) developed by HPCC which allows HPCC cluster to acess HDFS data

i have installed the connector but when i run the program to acess data from hdfs it gives error as libhdfs.so.0 doesn't exist.

I tried to build libhdfs.so using command

 ant compile-libhdfs -Dlibhdfs=1

its giving me error as

target "compile-libhdfs" does not exist in the project "hadoop"

i used one more command

 ant compile-c++-libhdfs -Dlibhdfs=1

its giving error as

ivy-download:
  [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar

  [get] To: /home/hadoop/hadoop-0.20.203.0/ivy/ivy-2.1.0.jar

  [get] Error getting http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
to /home/hadoop/hadoop-0.20.203.0/ivy/ivy-2.1.0.jar

BUILD FAILED java.net.ConnectException: Connection timed out

any suggestion will be a great help

Solution

Chhaya, you might not need to build libhdfs.so, depending on how you installed hadoop, you might already have it.

Check in HADOOP_LOCATION/c++/Linux-<arch>/lib/libhdfs.so, where HADOOP_LOCATION is your hadoop install location, and arch is the machine’s architecture (i386-32 or amd64-64).

Once you locate the lib, make sure the H2H connector is configured correctly (see page 4 here).

It's just a matter of updating the HADOOP_LOCATION var in the config file: /opt/HPCCSystems/hdfsconnector.conf

good luck.