Search code examples
bashawksedexinplace-editing

shell script replace variables in file - error with Sed's -i option for in-place updating


Here is my test.env

RABBITMQ_HOST=127.0.0.1
RABBITMQ_PASS=1234

And I want to use test.sh to replace the value in test.env to :

RABBITMQ_HOST=rabbitmq1
RABBITMQ_PASS=12345

here is my test.sh

#!/bin/bash
echo "hello world"

RABBITMQ_HOST=rabbitmq1
RABBITMQ_PASS=12345
Deploy_path="./config/test.env"

sed -i 's/RABBITMQ_HOST=.*/RABBITMQ_HOST='$RABBITMQ_HOST'/'  $Deploy_path
sed -i 's/RABBITMQ_PASS=.*/RABBITMQ_PASS='$RABBITMQ_HOST'/'  $Deploy_path 

But I have error

sed: 1: "./config/test.env": invalid command code .
sed: 1: "./config/test.env": invalid command code . 

How can I fix it?


Solution

  • tl;dr:

    With BSD Sed, such as also found on macOS, you must use -i '' instead of just -i (for not creating a backup file) to make your commands work; e.g.:

    sed -i '' 's/RABBITMQ_HOST=.*/RABBITMQ_HOST='"$RABBITMQ_HOST"'/'  "$Deploy_path"
    

    To make your command work with both GNU and BSD Sed, specify a nonempty option-argument (which creates a backup) and attach it directly to -i:

    sed -i'.bak' 's/RABBITMQ_HOST=.*/RABBITMQ_HOST='"$RABBITMQ_HOST"'/'  "$Deploy_path" &&
      rm "$Deploy_path.bak" # remove unneeded backup copy
    

    Background information, (more) portable solutions, and refinement of your commands can be found below.


    Optional Background Information

    It sounds like you're using BSD/macOS sed, whose -i option requires an option-argument that specifies the suffix of the backup file to create.
    Therefore, it is your sed script that (against your expectations) is interpreted as -i's option-argument (the backup suffix), and your input filename is interpreted as the script, which obviously fails.

    By contrast, your commands use GNU sed syntax, where -i can be used by itself to indicate that no backup file of the input file to updated in-place is to be kept.

    The equivalent BSD sed option is -i '' - note the technical need to use a separate argument to specify the option-argument '', because it is the empty string (if you used -i'', the shell would simply strip the '' before sed ever sees it: -i'' is effectively the same as just -i).

    Sadly, this then won't work with GNU sed, because it only recognizes the option-argument when directly attached to -i, and would interpret the separate '' as a separate argument, namely as the script.

    This difference in behavior stems from a fundamentally differing design decision behind the implementation of the -i option and it probably won't go away for reasons of backward compatibility.[1]

    If you do not want a backup file created, there is no single -i syntax that works for both BSD and GNU sed.

    There are four basic options:

    • (a) If you know that you'll only be using either GNU or BSD sed, construct the -i option accordingly: -i for GNU sed, -i '' for BSD sed.

    • (b) Specify a nonempty suffix as -i's option-argument, which, if you attach it directly to the -i option, works with both implementations; e.g., -i'.bak'. While this invariably creates a backup file with suffix .bak, you can just delete it afterward.

    • (c) Determine at runtime which sed implementation you're dealing with and construct the -i option accordingly.

    • (d) omit -i (which is not POSIX-compliant) altogether, and use a temporary file that replaces the original on success: sed '...' "$Deploy_path" > tmp.out && mv tmp.out "$Deploy_path".
      Note that this is in essence what -i does behind the scenes, which can have unexpected side effects, notably an input file that is a symlink getting replaced with a regular file; -i, does, however, preserve certain attributes of the original file: see the lower half of this answer of mine.

    Here's a bash implementation of (c) that also streamlines the original code (single sed invocation with 2 substitutions) and makes it more robust (variables are double-quoted):

    #!/bin/bash
    
    RABBITMQ_HOST='rabbitmq1'
    RABBITMQ_PASS='12345'
    Deploy_path="test.env"
    
    # Construct the Sed-implementation-specific -i option-argument.
    # Caveat: The assumption is that if the `sed` is not GNU Sed, it is BSD Sed,
    #         but there are Sed implementations that don't support -i at all,
    #         because, as Steven Penny points out, -i is not part of POSIX.
    suffixArg=()
    sed --version 2>/dev/null | grep -q GNU || suffixArg=( '' )
    
    sed -i "${suffixArg[@]}" '
     s/^\(RABBITMQ_HOST\)=.*/\1='"$RABBITMQ_HOST"'/
     s/^\(RABBITMQ_PASS\)=.*/\1='"$RABBITMQ_PASS"'/
    ' "$Deploy_path"
    

    Note that with the specific values defined above for $RABBITMQ_HOST and $RABBITMQ_PASS, it is safe to splice them directly into the sed script, but if the values contained instances of &, /, \, or newlines, prior escaping would be required so as not to break the sed command.
    See this answer of mine for how to perform generic pre-escaping, but you may also consider other tools at that point, such as awk and perl.


    [1] GNU Sed considers the option-argument to -i optional, whereas BSD Sed considers it mandatory, which is also reflected in the syntax specs. in the respective man pages: GNU Sed: -i[SUFFIX] vs. BSD Sed -i extension.