Search code examples
hadoopdistcp

Hadoop distcp not working

I am trying to copy data from one HDFS to another HDFS. Any suggestion why 1st one works but not 2nd one?

(works)

hadoop distcp hdfs://abc.net:8020/foo/bar webhdfs://def.net:14000/bar/foo

(does not work )

hadoop distcp webhdfs://abc.net:50070/foo/bar webhdfs://def:14000/bar/foo

Thanks!

Solution

  • If the two cluster are running incompatible version of HDFS, then you can use the webhdfsprotocol to distcp between them.

    hadoop distcp webhdfs://namenode1:50070/source/dir webhdfs://namenode2:50070/destination/dir

    NameNode URI and NameNode HTTP port should be provided in the source and destination command, if you are using webhdfs.