Search code examples
shellshposix

How should a backslash resulting from variable expansion be treated?

I ran the following command (*sh being the name of a sh implementation) with all the shells I could find; although I was expecting all to print match, I got inconsistent results. I don't know which behavior is correct and reliable.

*sh -c 'case "$1" in $2) echo match; esac' _ 'f\oo' 'f\\oo'

With dash from Ubuntu bionic's repo (and ash; which is a symbolic link to dash)

$ dash -c 'case "$1" in $2) echo match; esac' _ 'f\oo' 'f\\oo'
match

With bash 4.4.20(1)-release (x86_64-pc-linux-gnu) and 5.0.11(1)-release (arm-unknown-linux-androideabi)

$ bash -c 'case "$1" in $2) echo match; esac' _ 'f\oo' 'f\\oo'
match

With ksh Version AJM 93u+ 2012-08-01, and Version JM 93t+ 2010-03-05 (comes preinstalled with SunOS omniosce 5.11)

$ ksh -c 'case "$1" in $2) echo match; esac' _ 'f\oo' 'f\\oo'
match

With ksh @(#)PD KSH v5.2.14 99.07.13.2 (the default shell on OpenBSD 6.6, and its Linux port)

$ ksh -c 'case "$1" in $2) echo match; esac' _ 'f\oo' 'f\\oo'
$

With lksh @(#)LEGACY KSH R56 2018/01/14, mksh @(#)MIRBSD KSH R56 2018/01/14 (these are different binaries on Ubuntu bionic), and mksh @(#)MIRBSD KSH R57 2019/03/01

$ lksh -c 'case "$1" in $2) echo match; esac' _ 'f\oo' 'f\\oo'
$ mksh -c 'case "$1" in $2) echo match; esac' _ 'f\oo' 'f\\oo'
$

With posh 0.13.1

$ posh -c 'case "$1" in $2) echo match; esac' _ 'f\oo' 'f\\oo'
$

With yash 2.46

$ yash -c 'case "$1" in $2) echo match; esac' _ 'f\oo' 'f\\oo'
$

And with zsh 5.4.2 (x86_64-ubuntu-linux-gnu) and 5.7.1 (arm-unknown-linux-androideabi); emulating sh

$ zsh -c 'emulate sh; case "$1" in $2) echo match; esac' _ 'f\oo' 'f\\oo'
match

And I'm lost in POSIX's Shell Command Language specification†; couldn't find a straightforward answer to my question yet: How should an escaped backslash resulting from a variable expansion be interpreted in a glob pattern? As \\ or as \? Or is it unspecified?

† Under Case Conditional Construct it says:

In order from the beginning to the end of the case statement, each pattern that labels a compound-list shall be subjected to tilde expansion, parameter expansion, command substitution, and arithmetic expansion, and the result of these expansions shall be compared against the expansion of word, according to the rules described in Pattern Matching Notation (which also describes the effect of quoting parts of the pattern)

Notice it doesn't say patterns are subjected to quote removal; but under Pattern Matching Notation, it says:

A <backslash> character shall escape the following character. The escaping <backslash> shall be discarded

But it doesn't clarify if that still happens when the pattern is a result of expansion.

For anyone who's interested, the standard is unclear on this; they'll amend it in newer versions though. Below are some links to POSIX bug reports where this issue was discussed broadly.

Solution

  • Perhaps this bit of reference can clear things up for your question. From the POSIX's Shell Command Language specification - under Pattern Matching Notation the clause states,

    2.13.1 Patterns Matching a Single Character

    A <backslash> character shall escape the following character. The escaping <backslash> shall be discarded. If a pattern ends with an unescaped <backslash>, it is unspecified whether the pattern does not match anything or the pattern is treated as invalid

    The above rule applies to patterns while using the case statements and when using the the == glob match operator in bash using test construct.

    So since your $2 remains unquoted while undergoing pattern match, the literal value of f\\oo is lost forever and gets interpreted as f\oo.

    Just to clear things up, the shell has preserved your literal value during quote removal and just when applying the passed argument in this pattern matching rule, this behavior is exhibited.