I have seen GNU parallel with rsync, unfortunately, I cannot see a clear answer for my use case.
As part of my script I have this:
echo "file01.zip
file02.zip
file03.zip
" | ./gnu-parallel --line-buffer --will-cite \
-j 2 -t --verbose --progress --interactive \
rsync -aPz {} [email protected]:/home/user/
So, I run the script, and as a part of its output, once it gets to the gnu-parallel step, I get this (because I have --interactive, I get prompted to confirm each file:
rsync -aPz file01.zip [email protected]:/home/user/ ?...y
rsync -aPz file02.zip [email protected]:/home/user/ ?...y
Computers / CPU cores / Max jobs to run
1:local / 4 / 2
Computer:jobs running/jobs completed/%of started jobs/Average seconds to complete
local:2/0/100%/0.0s
... and then, the process just hangs here and does nothing; no numbers change or anything.
At this point, I can do from another terminal this:
$ ps axf | grep rsync
12754 pts/1 S+ 0:00 | | \_ perl ./gnu-parallel --line-buffer --will-cite -j 2 -t --verbose --progress --interactive rsync -aPz {} [email protected]:/home/user/
12763 pts/1 T 0:00 | | \_ rsync -aPz file01.zip [email protected]:/home/user/
12764 pts/1 R 0:11 | | | \_ ssh -l user example.com rsync --server -logDtprze.iLs --log-format=X --partial . /home/user/
12766 pts/1 T 0:00 | | \_ rsync -aPz file02.zip [email protected]:/home/user/
12769 pts/1 R 0:10 | | \_ ssh -l user example.com rsync --server -logDtprze.iLs --log-format=X --partial . /home/user/
... and so I can indeed confirm that processes have been started, but they are apparently not doing anything. As to confirmation that they are not doing anything (as opposed to uploading, which they should be doing in this case), I ran the monitor sudo iptraf, and it reported 0 Kb/s for all traffic on wlan0, which is the only one I have here.
The thing is - the server where I'm logging in to, accepts only SSH authentication with passwords. At first I thought --interactive would allow me to enter the passwords interactively, but instead it prompts the user about whether to run each command line and read a line from the terminal. Only run the command line if the response starts with 'y' or 'Y'.. So ok, above I answered y, but it doesn't prompt me for a password afterwards, and it seems the processes are hanging there waiting for it. My version is "GNU parallel 20160422".
$ ./gnu-parallel --version | head -1
GNU parallel 20160422
So, how can I use GNU parallel, to run multiple rsync tasks with passwords?
Use sshpass:
doit() {
rsync -aPz -e "sshpass -p MyP4$$w0rd ssh" "$1" [email protected]:/home/user
}
export -f doit
parallel --line-buffer -j 2 --progress doit ::: *.zip
The fundamental problem with running interactive programs in parallel is: which program should get the input if two programs are ready for input? Therefore GNU Parallel's --tty implies -j1.