Having a problem where running a GNU Parallel job in distributed mode (ie. across multiple machines via the --sshloginfile) and finding that even though the job is running on each machine as the same user (or at least dictated that way in the file being given to the --sshloginfile (eg. myuser@myhostname00x)), getting a "Permission denied" error when the job tries to access a file. This occurs despite being able to (passwordless) ssh into the remote nodes in question and ls the files that the Parallel job claims it has no permissions for (the specified path is to a filesystem that is shared and NFS mounted on all the nodes).
Have a list file of nodes like
me@host001
me@host005
me@host006
and the actual Parallel job looks like
bcpexport() {
<do stuff to arg $1 to BCP copy to a MSSQL DB>
}
export -f bcpexport
parallel -q -j 10 --sshloginfile $basedir/src/parallel-nodes.txt --env $bcpexport \
bcpexport {} "$TO_SERVER_ODBCDSN" $DB $TABLE $USER $PASSWORD $RECOMMEDED_IMPORT_MODE $DELIMITER \
::: $DATAFILES/$TARGET_GLOB
where the $DATAFILES/$TARGET_GLOB glob pattern returns files from a directory. Running this job in single node mode works fine, but when running across all the nodes in the parallel-nodes.txt file throws
/bin/bash: line 27: /path/to/file001: Permission denied
/bin/bash: line 27: /path/to/file002: Permission denied
...and so on for all the files...
If anyone knows what could be going on here, advice or debugging suggestions would be appreciated.
I think the problem is the additional $:
parallel [...] --env $bcpexport bcpexport {} [...]
Unless you set the shell variable $bcpexport to something you probably meant bcpexport (no $) instead.
If $bcpexport is undefined, then it will be replace with nothing by the shell. Thus --env will eat the next argument, so you will really be running:
parallel [...] --env bcpexport {} [...]
which will execute {} as a command, which is exactly what you experience.
So try this instead:
parallel [...] --env bcpexport bcpexport {} [...]