I am getting the following different behaviors for the commands when run locally vs that when run remotely via ssh.
$ /bin/csh -c \'/bin/echo a b c d\'
Unmatched '''.
$ ssh node1 /bin/csh -c \'/bin/echo a b c d\'
a b c d
Does ssh here is handling the command arguments differently and removing the escape chars somehow? (or the local shell itself is doing that?)
TLDR: Your example is failing because of local shell processing.
On Unix when process x (such as a shell) runs a different program y (usually as a child process, but not necessarily) it passes y's own name plus any arguments as a variable length array of strings, e.g. on an old system echo a b c d
runs the echo
program with the array of arguments (shown one per line for clarity)
echo
a
b
c
d
(On most shells on recent system echo
is shell builtin and not a separate program, but other programs still work the traditional way.)
Because C-arrays start at subscript 0 and the argument array is conventionally named argv
(argument vector -- C derived from BCPL where arrays were called vectors) C (and C++ etc) programmers often call the program name "argv-zero". Note that you can have the same program appear under multiple names in the filesystem, so its name isn't known at compile time, only when it is run; this trick used to be quite popular for things like gzip/gunzip
but in recent decades it seems to have become unfashionable.
In shell input a backslashed singlequote means to treat that singlequote as ordinary data, not use it to quote other characters such as a space the delimits words and thus arguments, so
/bin/csh -c \'/bin/echo a b c d\'
runs /bin/csh
with the following argv:
/bin/csh
-c
'/bin/echo
a
b
c
d'
csh (or a Bourne-type shell) takes one argument after -c
as an entire command line (or script) to be parsed and the result (possibly) executed, but '/bin/echo
is not a valid command line. Any remaining arguments are not considered as commands but as possible arguments to the command being run if it needs them.
The ssh
client program on the other hand treats any and all 'data' arguments (not options and not the required [user@]remotehost argument) as forming the command to be run remotely, and sends them over the SSH protocol all concatenated into one line, so your ssh
command gets
ssh
node1
/bin/csh
-c
'/bin/echo
a
b
c
d'
but sends
/bin/csh -c '/bin/echo a b c d'
and sshd
on the remote system passes that entire line as one argument, preceded by a separate -c
argument, to your login shell, which then runs csh with
/bin/csh
-c
/bin/echo a b c d
so it takes the one argument after -c as a command line, which it can parse successfully and run.
And this combination is why we have dozens of Qs on several Stacks for many variations of "how can I get ssh to remotely run this command using special characters and/or interpolations correctly". While there is usually a possible answer, it is often easiest to punt either by running ssh with no remote command argument(s), so it remotely runs an interactive shell, and sending the desired command(s) as input to that shell; or, almost the same, putting the command(s) in a file, transferring that file to the remote system (or making it accessible via NFS or similar) and then telling the remote system to run the file as a script.