Search code examples
shellposix

does quote removal happen after command substitution in a POSIX shell?


The POSIX shell standard at

http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07_04

says in Section 2.6:

command substitution (...) shall be performed

(...)

Quote removal (...) shall always be performed last.

It appears to me that quote removal is not performed after command substitution:

$ echo "#"
#
$ echo '"'
"

as expected, but

$ echo $(echo '"')#"
>

What am I not understanding?

Added after reading answer/comments:

From what everybody is saying, the consideration of quotes happens at the very beginning of parsing, for example, to decide if a command is even "acceptable". Then why does the standard bother to emphasise, that the quote removal is performed late in the process??


Solution

  • "then the outer command becomes echo "#" and is balanced"

    That is not 'balanced' because the first double-quote does not count. Quotes are only meaningful as quotes if they appear unencumbered on the command line.

    To verify, let's look at this:

    $ echo $(echo '"')#
    "#
    

    That is balanced because the shell does considers that " to be just another character.

    By contrast, this is unbalanced because it has one and only one shell-active ":

    $ echo $(echo '"')#"
    > 
    

    Similar example 1

    Here we show the same thing but using parameter expansion instead of command substitution:

    $ q='"'; echo $q
    "
    

    Once the shell has substituted " for $q, one might think that there was an unbalanced double-quote. But, that double-quote was the results of parameter expansion and is therefore not a shell-active quote.

    Similar example 2

    Let's consider a directory containing file:

    $ ls 
    file
    $ ls "file"
    file
    

    As you can see above, quote removal is perfomed before ls is run.

    But, consider this command:

    $ echo ls $(echo '"file"')
    ls "file"
    

    As you can see ls $(echo '"file"') expands to ls "file" which is the command which ran successfully above. Now, let's try running that:

    $ ls $(echo '"file"')
    ls: cannot access '"file"': No such file or directory
    

    As you can see, the shell does not treat the double-quotes that remain after command substitution. This is because those quotes are not considered to be shell-active. As a consequence, they are treated as normal characters and passed on to ls which complains that the file whose name begins and ends with " does not exist in the directory.

    The same is happening here:

    $ cmd='ls "file"'
    $ $cmd
    ls: cannot access '"file"': No such file or directory
    

    POSIX standard

    From the POSIX standard:

    Enclosing characters in single-quotes ( ' ' ) shall preserve the literal value of each character within the single-quotes

    In other words, once the double-quote appears inside single quotes, it has no special powers: it is just another character.

    The standard also mentions escaping and double-quotes as methods of preserving "the literal value" of a character.

    Practical consequences

    People new to shell often want to store a command in a variable as in the cmd='ls "file"' example above. But, because quotes and other shell-active characters cease to be shell active once they are stored in a variable, the complex cases always fail. This leads to a classic essay:

    "I'm trying to put a command in a variable, but the complex cases always fail!"