IFS and command substitution

Question

I am writing a shell script to read input csv files and run a java program accordingly.

#!/usr/bin/ksh
CSV_FILE=${1}
myScript="/usr/bin/java -version"
while read row
do
     $myScript
     IFS=$"|"
     for column in $row
     do 
        $myScript
     done
done < $CSV_FILE

csv file:

a|b|c

Interestingly, $myScript outside the for loop works but the $myScript inside the for loop says "/usr/bin/java -version: not found [No such file or directory]". I have come to know that it is because I am setting IFS. If I comment IFS, and change the csv file to

a b c

It works ! I imagine the shell using the default IFS to separate the command /usr/bin/java and then apply the -version argument later. Since I changed the IFS, it is taking the entire string as a single command - or that is what I think is happening.

But this is my requirement: I have a csv file with a custom delimiter, and the command has arguments in it, separated by space. How can I do this correctly?

Answer 1

IFS indicates how to split the values of variables in unquoted substitutions. It applies to both $row and $myscript.

If you want to use IFS to do the splitting, which is convenient in plain sh, then you need to change the value of IFS or arrange to need the same value. In this particular case, you can easily arrange to need the same value, by defining myScript as myScript="/usr/bin/java|-version". Alternatively, you can change the value of IFS just in time. In both cases, note that an unquoted substitution doesn't just split the value using IFS, it also interprets each part as a wildcard pattern and replaces it by the list of matching file names if there are any. This means that if your CSV file contains a line like

foo|*|bar

then the row won't be foo, *, bar but foo, each file name in the current directory, bar. To process the data like this, you need to turn off with set -f. Also remember that read reads continuation lines when a line ends with a backslash, and strips leading and trailing IFS characters. Use IFS= read -r to turn off these two behaviors.

myScript="/usr/bin/java -version"
set -f
while IFS= read -r row
do
    $myScript
    IFS='|'
    for column in $row
    do 
        IFS=' '
        $myScript
    done
done

However there are better ways that avoid IFS-splitting altogether. Don't store a command in a space-separated string: it fails in complex cases, like commands that need an argument that contains a space. There are three robust ways to store a command:

• Store the command in a function. This is the most natural approach. Running a command is code; you define code in a function. You can refer to the function's arguments collectively as "$@".

  myScript () {
      /usr/bin/java -version "$@"
  }
  …
  myScript extra_argument_1 extra_argument_2
  

• Store an executable command name and its arguments in an array.

  myScript=(/usr/bin/java -version)
  …
  "${myScript[@]}" extra_argument_1 extra_argument_2
  

• Store a shell command, i.e. something that is meant to be parsed by the shell. To evaluate the shell code in a string, use eval. Be sure to quote the argument, like any other variable expansion, to avoid premature wildcard expansion. This approach is more complex since it requires careful quoting. It's only really useful when you have to store the command in a string, for example because it comes in as a parameter to your script. Note that you can't sensibly pass extra arguments this way.

  myScript='/usr/bin/java -version'
  …
  eval "$myScript"
  

Also, since you're using ksh and not plain sh, you don't need to use IFS to split the input line. Use read -A instead to directly split into an array.

#!/usr/bin/ksh
CSV_FILE=${1}
myScript=(/usr/bin/java -version)
while IFS='|' read -r -A columns
do
    "${myScript[@]}"
    for column in "${columns[@]}"
    do 
        "${myScript[@]}"
    done
done <"$CSV_FILE"