Search code examples
hadoophdfsscp

Copying a directory from a remote HDFS local file system to my local machine

I have a directory on my local hdfs environment, I want to copy it to my local computer. I am accessing the hdfs using ssh (with a password).

I tried many suggested copy command but did not work.

What I tried: 

scp ‘username@hn0-sc-had:Downloads/*’ ~/Downloads

as mentioned in this link. What am I doing wrong?

Solution

  • SCP will copy from the remote Linux server.

    HDFS does not exist on a single server or is a "local filesystem", therefore SCP is not the right tool to copy from it directly

    Your options include

    1. SSH to remote server
    2. Use hdfs dfs -copyToLocal in order to pull files from HDFS
    3. Use SCP from your computer to get the files you just downloaded on the remote server

    Or

    1. Configure a local Hadoop CLI using XML files from remote server
    2. Use hdfs dfs -copytoLocal directly against HDFS from your own computer

    Or

    1. Install HDFS NFS Gateway
    2. Mount an NFS volume on your local computer, and copy of files from it