Search code examples
linuxbashshellscpgnu-parallel

copy small files in parallel from remote servers

I have to copy lots of files (around 2000 file and each file size around 5MB) from remote servers on a local box so I usually run below command on my local box terminal to copy files:

scp goldy@machineA:/process/snap/20180418/* /data/files/

Is there any way to copy those files in parallel from remote servers using gnu-parallel or anything else? I mean something like copy 5 files in parallel instead of doing everything one by one.

Note: below is my file names on remote servers.

abc_proc_1_log.data
abc_proc_2_log.data
abc_proc_3_log.data
abc_proc_4_log.data
...
abc_proc_2000_log.data
Solution 
  • ssh goldy@machineA 'ls -1 /process/snap/20180418/*' \
| parallel -j5 'scp goldy@machineA:{} /data/files/{}'
    • ssh goldy@machineA 'ls -1 /process/snap/20180418/*' - get filename list from the remote machine beforehand
    • parallel -j5 ... - run up to 5 jobs in parralel