Search code examples
bashshellawk

How do I use shell variables in an awk script?

I found some ways to pass external shell variables to an awk script, but I'm confused about ' and ".

First, I tried with a shell script:

$ v=123test
$ echo $v
123test
$ echo "$v"
123test

Then tried awk:

$ awk 'BEGIN{print "'$v'"}'
$ 123test
$ awk 'BEGIN{print '"$v"'}'
$ 123

Why is the difference?

Lastly I tried this:

$ awk 'BEGIN{print " '$v' "}'
$  123test
$ awk 'BEGIN{print ' "$v" '}'
awk: cmd. line:1: BEGIN{print
awk: cmd. line:1:             ^ unexpected newline or end of string

I'm confused about this.

Solution

  • #Getting shell variables into awk may be done in several ways. Some are better than others. This should cover most of them. If you have a comment, please leave below.                                                                                    v1.5

    Using -v (The best way, most portable)

    Use the -v option: (P.S. use a space after -v or it will be less portable. E.g., awk -v var= not awk -vvar=)

    variable="line one\nline two"
awk -v var="$variable" 'BEGIN {print var}'
line one
line two

    This should be compatible with most awk, and the variable is available in the BEGIN block as well:

    If you have multiple variables:

    awk -v a="$var1" -v b="$var2" 'BEGIN {print a,b}'

    Warning. As Ed Morton writes and as seen in the above example, the shell variable is expanded by the shell before awk then sees its content as awk -v var='line one\nline two' and so any escape sequences in the content of that shell variable will be interpreted when using -v, just like they are for every other form of assignment of a string to a variable in awk, e.g. awk 'BEGIN{var="line one\nline two"} {...}' or awk '{...}' var='line one\nline two', and so \n becomes a literal LineFeed character and not the 2-character string \n. For example, given a variable like:

    $ variable='a\tb\n$c\kd'

    awk would expand the escape sequences in the assignment:

    $ awk -v var="$variable" 'BEGIN{ printf "%s\n", var }'
awk: warning: escape sequence `\k' treated as plain `k'
a       b
$ckd

    If that's not what you want then, if your shell (e.g. bash) and locale (e.g. LC_ALL=C) support it then you can have backslashes treated literally by using shell parameter substitution to escape any backslashes:

    $ awk -v var="${variable//\\/\\\\}" 'BEGIN{ printf "%s\n", var }'
a\tb\n$c\kd

    or by using ENVIRON[] or access it via ARGV[] (see below).

    You cannot use -v var="$(printf '%q' "$variable")" for this as that would also escape $s, nor can you use -v var="${variable@Q}" as that would just add 's around "$variable" and the escape sequences would still be interpreted by awk. That's because those 2 approaches both escape chars according to shell syntax for providing command input, not awk syntax for assigning strings to variables.

    PS If you have vertical bar or other regexp meta characters as separator like |?( etc, they must be double escaped. Example 3 vertical bars ||| becomes -F'\\|\\|\\|'. You can also use -F"[|][|][|]".

    Example on getting data from a program/function in to awk (here date is used)

    awk -v time="$(date +"%F %H:%M" -d '-1 minute')" 'BEGIN {print time}'

    Example of testing the contents of a shell variable as a regexp:

    awk -v var="$variable" '$0 ~ var{print "found it"}'

    Variable after code block

    Here we get the variable after the awk code. This will work fine as long as you do not need the variable in the BEGIN block:

    variable="line one\nline two"
echo "input data" | awk '{print var}' var="${variable}"
or
awk '{print var}' var="${variable}" file
    • Adding multiple variables:

    awk '{print a,b,$0}' a="$var1" b="$var2" file

    • In this way we can also set different Field Separator FS for each file.

    awk 'some code' FS=',' file1.txt FS=';' file2.ext

    • Variable after the code block will not work for the BEGIN block:

    echo "input data" | awk 'BEGIN {print var}' var="${variable}"

    Here-string

    Variable can also be added to awk using a here-string from shells that support them (including Bash):

    awk '{print $0}' <<< "$variable"
test

    This is the same as:

    echo "$variable" | awk '{print $0}'
printf '%s' "$variable" | awk '{print $0}'

    P.S. this treats the variable as a file input.

    ENVIRON input

    As TrueY writes, you can use the ENVIRON to print Environment Variables. Setting a variable before running AWK, you can print it out like this:

    export X=MyVar
awk 'BEGIN{print ENVIRON["X"],ENVIRON["SHELL"]}'
MyVar /bin/bash

    or for a non-exported variable:

    x=MyVar
x="$x" awk 'BEGIN{print ENVIRON["x"],ENVIRON["SHELL"]}'
MyVar /bin/bash

    ARGV input

    As Steven Penny writes, you can use ARGV to get the data into awk:

    v="my data"
awk 'BEGIN {print ARGV[1]}' "$v"
my data

    To get the data into the code itself, not just the BEGIN:

    v="my data"
echo "test" | awk 'BEGIN{var=ARGV[1];ARGV[1]=""} {print var, $0}' "$v"
my data test

    Variable within the code: USE WITH CAUTION

    You can use a variable within the awk code, but it's messy and hard to read, and as Charles Duffy points out, this version may also be a victim of code injection. If someone adds bad stuff to the variable, it will be executed as part of the awk code.

    This works by extracting the variable within the code, so it becomes a part of it.

    If you want to make an awk that changes dynamically with use of variables, you can do it this way, but DO NOT use it for normal variables.

    variable="line one\nline two"
awk 'BEGIN {print "'"$variable"'"}'
line one
line two

    Here is an example of code injection:

    variable='line one\nline two" ; for (i=1;i<=1000;++i) print i"'
awk 'BEGIN {print "'"$variable"'"}'
line one
line two
1
2
3
.
.
1000

    You can add lots of commands to awk this way. Even make it crash with non valid commands.

    One valid use of this approach, though, is when you want to pass a symbol to awk to be applied to some input, e.g. a simple calculator:

    $ calc() { awk -v x="$1" -v z="$3" 'BEGIN{ print x '"$2"' z }'; }

$ calc 2.7 '+' 3.4
6.1

$ calc 2.7 '*' 3.4
9.18

    There is no way to do that using an awk variable populated with the value of a shell variable, you NEED the shell variable to expand to become part of the text of the awk script before awk interprets it. (see comment below by Ed M.)

    Extra info:

    Use of double quote

    It's always good to double quote variable "$variable"
    If not, multiple lines will be added as a long single line.

    Example:

    var="Line one
This is line two"

echo $var
Line one This is line two

echo "$var"
Line one
This is line two

    Other errors you can get without double quote:

    variable="line one\nline two"
awk -v var=$variable 'BEGIN {print var}'
awk: cmd. line:1: one\nline
awk: cmd. line:1:    ^ backslash not last character on line
awk: cmd. line:1: one\nline
awk: cmd. line:1:    ^ syntax error

    And with single quote, it does not expand the value of the variable:

    awk -v var='$variable' 'BEGIN {print var}'
$variable

    More info about AWK and variables

    Read this faq.