Usage of envsubst combined with tee randomly results in an empty file

Question

I'm looking to understand what is going on in a simple script, that seems to produce random results.

What I am trying to do:

• Replace variables in pre-existing files from the values defined in the environment.

• This is done inside a Docker container with a bash script, that runs the command:

  envsubst '$VAR1 $VAR2' < $FILE | tee $FILE

---

What happens:

• Sometimes a $FILE in question has contents before the command, but contains nothing after the command.

---

How to reproduce the issue:

• Dockerfile:

FROM debian:stretch

RUN apt-get update -qy
RUN apt-get install -qy gettext

COPY main-script /main-script
RUN chmod +x /main-script

ENTRYPOINT [ "/main-script" ]

• Bash script:

#!/bin/bash

mkdir -p /test

export TEST1=1
export TEST2=2
export TEST3=3

for I in {1..300} ; do
    echo '$TEST1 $TEST2 $TEST3' > /test/file-$I
done

for FILE in /test/file-* ; do
    envsubst < $FILE | tee $FILE
done

for FILE in /test/file-* ; do
    if [[ -z "$(cat $FILE)" ]]; then
        echo "$FILE is empty!"
        FAIL=1
    fi
done

if [[ -n "$FAIL" ]]; then
    exit 2
fi

Output looks something like this:

...
/test/file-11 is empty!
/test/file-180 is empty!
/test/file-183 is empty!
/test/file-295 is empty!

Answer 1

Pipes are asynchronous, and you've introduced a race condition. You can't predict if envsubst reads from $FILE before or after tee truncates it.

The correct approach is to write the changes to a temporary file, then replace the original with the temporary file after that has succeeded.

tmp=$(mktemp)
envsubst < "$FILE" > "$tmp" &&  mv "$tmp" "$FILE"