Search code examples
bashshellsh

Make sed to replace capture group with corresponding environment variable

Assuming we have source_file.json of the following content:

 {
   "someVar": "__SOME_ENV_VAR__",
   "anotherVar": "__ANOTHER_ENV_VAR__",
   "yetAnotherVar": "__ONE_MORE_ENV_VAR__"
 }

How do I

  • Search for environment variable placeholder (using the pattern __([A-Z_]\+)__
  • Interpolate capturing group (([A-Z_]\+) part) as \1 to the value of the environment variable that matches the capturing group (i.e. SOME_ENV_VAR, ANOTHER_ENV_VAR, ONE_MORE_ENV_VAR)
  • Replace the capturing group with the value of the corresponding environment variable

I tried to approach this problem from two different angles (with numerous variations)

No1: search and replace in one go using sed

sed -i "s/__([A-Z_]\+)__/${\1}/g" source_file.json

No2 (worse one): search and replace all env var occurrences

env | while IFS='=' read -r key value; do
  sed -i "s/__$key__/$value/g"
done

But none of them did particularly work all failing with different kinds of errors or coming short of interpolating the environment variable and outputting something like "someVar": "$SOME_ENV_VAR".

None of the posts related to the subject cover this particular case (interpolation of a capturing group). Most of them cover the scenario with specific environment variables to be interpolated known ahead of time.

Note: above has to go into /bin/sh script that has to be executed inside Alpine-based docker container and thus no hassle with basetext package installation (to use envsubst) considered as an option

Solution

  • If the /bin/sh of the image is Bash, then you could use a loop to build a Bash array with the replacement commands, and then call sed:

    replacements=()
while IFS='=' read -r key value; do
  replacements+=(-e "s/__${key}__/$value/")
done < <(env)

sed -i "${replacements[@]}" /path/to/file

    If it is not Bash, then you could create a sed script with a loop, and then pass that to sed (thanks @tripleee for the nudge):

    script=/tmp/sed.txt

env | while IFS='=' read -r key value; do
  echo "s/__${key}__/$value/"
done > "$script"

sed -f "$script" -i /path/to/file

    Both approaches have the weakness that they break if the value (or the key) contains the pattern and replace separator / in the s/// command of sed. If there is a symbol that is guaranteed to be not used by any key and value, then you can use that. For example instead of s///, the command can be written as s???.