POSIX shell: escaping line-continuations in backquote command-substitutions

Question

I'm writing a shell, and I'm a little confused by the POSIX shell specification. Say I have the command:

echo "`echo "a\\
b"`"

Should the shell output

ab

or

a\
b

?

In other words, are line continuations removed again after removing the escaping from the text in a command substitution? The POSIX specification appears to specify that line-continuation removal will not happen again, however, all the shells I tested (bash, dash, and busybox's ash) run line-continuation removal again, causing the test script to output ab.

Script explanation:

The part of the script that's inside the command-substitution is un-escaped, producing:

echo "a\
b"

Now, if line-continuation removal is run again, it will remove the backslash-newline pair, producing the command echo "ab" inside the command-substitution, otherwise the backslash-newline pair will still be between the a and b.

Answer 1

• Old-style `...` command substitutions subject the embedded command to prior interpretation of \ as an escape character, and only then parse and execute it.

  • This passage from the POSIX shell spec is key (edited for readability):

  Within the backquoted style of command substitution, \ shall retain its literal meaning, except when followed by: $, `, or \.

  • In other words: any embedded \$, \`, and \\ sequences are treated as escape sequences whose 2nd character should be treated literally.

  • Thus, \\<newline> in your command is reduced to \<newline>, because `...` interprets the \\ as an escaped, literal \

  • This interpretation happens before the embedded command is parsed and executed.

  • The \<newline> in the resulting command is therefore interpreted as a line continuation (inside the double-quoted string), which effectively removes the newline.

  • Therefore, the double-quoted string is effectively parsed as literal ab, and that is what is passed to the inner echo call.

  • In bash, you can verify this processing by setting debugging options: set -xv

• Modern syntax $(...) avoids such surprises by providing a truly independent quoting context.

  • To quote from the "Rationale for Shell and Utilities" part of the spec (emphasis added):

  Because of these inconsistent behaviors, the backquoted variety of command substitution is not recommended for new applications that nest command substitutions or attempt to embed complex scripts.

  • With $(...), the escaped line continuation in the embedded double-quoted string is retained (in bash, dash, ksh and zsh):

    echo "$(echo "a\\
    b")"
    
    # Output
    a\
    b         
    

  • Another reason to prefer $(...) is that it works the same in bash, dash, ksh and zsh, which is not true of `...`, whose behavior differs in ksh (see below).

---

Compliance in major POSIX-like shells - bash, dash, ksh, zsh

• In ksh (verified with version 93u+), your command breaks, because ksh requires embedded " chars. inside `...` to be escaped as \" - which is a deviation from the standard.
  Syntax $(...) does not have this requirement.

• bash, dash, and zsh process your `...`-based command as required by the spec (in the case of bash, whether or not it is run in POSIX-compatibility mode).

  • Note that these shells also support \"-escaped as double quotes inside `...` as ksh requires.
  • Arguably, supporting this is a deviation from the standard, given that " is not among the characters that form an escape sequence when preceded by \ in the context of `...`; e.g., echo "`echo \"a b\"`" should result in "a b", not a b.

---

Optional reading: cross-shell testing

If you find yourself needing to compare the behavior of POSIX-like shells frequently, consider use of shall, my CLI and REPL for invoking shell scripts or commands with multiple POSIX-like shells.

By default, it targets bash, dash, ksh, and zsh (whichever ones are installed).

If you put your command in script ./tst, for instance, you would invoke shall as follows:

shall ./tst

which yields something like:

Note how invocation with ksh failed, because ksh requires " inside a `...` command substitution to be escaped as \".
Again, using $(...) would bypass this problem.

Installation of shall from the npm registry (Linux and macOS)

^{Note: Even if you don't use Node.js, npm, its package manager, works across platforms and is easy to install; try
curl -L https://git.io/n-install | bash}

With Node.js installed, install as follows:

[sudo] npm install shall -g

Note:

• Whether you need sudo depends on how you installed Node.js and whether you've changed permissions later; if you get an EACCES error, try again with sudo.
• The -g ensures global installation and is needed to put shall in your system's $PATH.

Manual installation (any Unix platform with bash)

• Download this bash script as shall.
• Make it executable with chmod +x shall.
• Move it or symlink it to a folder in your $PATH, such as /usr/local/bin (macOS) or /usr/bin (Linux).