aws cloudformation deploy and --parameter-overrides syntax

Question

I'm using aws cli v1 (aws-cli/1.29.8 Python/3.9.17 Linux/4.14.291-218.527.amzn2.x86_64 exec-env/AWS_ECS_EC2 botocore/1.31.) within CodeBuild, and reading the documentation for the --parameter-overrides that is passed to the aws cloudformation deploy command it says:

--parameter-overrides (list) A list of parameter structures that specify input parameters for your stack template. If you're updating a stack and you don't specify a parameter, the command uses the stack's existing value. For new stacks, you must specify parameters that don't have a default value. Syntax: ParameterKey1=ParameterValue1 ParameterKey2=ParameterValue2 ...(string)

So basically, i can just pass the --parameter-overrides like:

> aws cloudformation deploy --parameter-overrides Key1=value1 Key2=value2

# or

> aws cloudformation deploy --parameter-overrides Key1=value1 \
Key2=value2

# or

> aws cloudformation deploy --parameter-overrides \
Key1=value1 \
Key2=value2

I understand that I can also quote each key=value pair like 'Key1=value1' (or use double quote).

I have the following json file for my parameters:

[
  {
    "ParameterKey": "Key1",
    "ParameterValue": "value1"
  },
  {
    "ParameterKey": "Key2",
    "ParameterValue": "$ENV_VALUE2"
  }
]

Both Key1 and Key2 are required in my CloudFormation template (have no default value specified)

I use the following jq command to convert to the --parameter-overrides syntax:

> export ENV_VALUE2=value2

> parameter_overrides="$(jq -r '.[] | .ParameterValue |= (env[.[1:]] // .) | [.ParameterKey, .ParameterValue] | join("=")' myfile.json)"

> echo "$parameter_overrides"
Key1=value1
Key2=value2

But running the command aws cloudformation deploy --parameter-overrides "$parameter_overrides" results in the following error:

An error occurred (ValidationError) when calling the CreateChangeSet operation: Parameters: [Key2] must have values

Maybe, this is happening because with the jq conversion i don't have a trailling \ to escape the line break, so I did a few tests.

test1

> export parameter_overrides=Key1=value Key2=value2
> aws cloudformation deploy --parameter-overrides "$parameter_overrides"

> An error occurred (ValidationError) when calling the CreateChangeSet operation: Parameters: [Key2] must have values

The same happens if i try to export with single and double quotes, like:

> export parameter_overrides="Key1=value Key2=value2"
or
> export parameter_overrides='Key1=value Key2=value2'

test2

> export parameter_overrides="'Key1=value' 'Key2=value2'"
> aws cloudformation deploy --parameter-overrides "$parameter_overrides"

> An error occurred (ValidationError) when calling the CreateChangeSet operation: Parameters: [Key2, Key2] must have values

The same happens if I swap the quotes.

test3

> export parameter_overrides="'Key1=value Key2=value2'"
> aws cloudformation deploy --parameter-overrides "$parameter_overrides"

> An error occurred (ValidationError) when calling the CreateChangeSet operation: Parameters: [Key2, Key2] must have values

The same happens if I swap the quotes.

test4

As the last resource, I tried to pass a json string:

export parameter_overrides='[ "Key1=value1", "Key2=value2" ]'

And the deploy just worked.

I can change my jq command to convert from one format to the other, but I'm really curious why everything other than passing a json string didn't worked as expected.

EDITED:

Based on the answer from @knittl, I'm using the following as a solution:

parameter_overrides=$(jq -r '.[] | .ParameterValue |= (env[.[1:]] // .) | [.ParameterKey, .ParameterValue] | join("=") | @sh' myfile.json | tr "\n" " ")

echo "$parameter_overrides" | xargs aws cloudformation deploy \
    --template-file 'my_cfn_stack.yml' \
    --stack-name 'my-stack-name' \
    --capabilities 'CAPABILITY_NAMED_IAM' 'CAPABILITY_AUTO_EXPAND' \
    --no-fail-on-empty-changeset \
    --parameter-overrides

I just discovered that jq have the function/operator @sh which will quote the output. Without the tr, the result of the $override_parameters variable is:

> echo "$override_parameters"
'Key1=value1'
'Key2=value2'

Using the tr "\n" " " results, worked like a charm:

> echo "$override_parameters"
'Key1=value1' 'Key2=value2'

But without the xargs it still doesn't works. Thank you @knittl

Answer 1

Parameter expansion + word/field splitting of your shell is biting you.

aws cloudformation deploy --parameter-overrides "$parameter_overrides"

is equivalent to:

aws cloudformation deploy --parameter-overrides 'key1=value1 key2=value2'

which is very different from

aws cloudformation deploy --parameter-overrides 'key1=value1' 'key2=value2'

(1 override vs 2 overrides)

Unfortunately, that's how shells expand parameters.

Not quoting the parameter would perform field splitting, but at the same time, pathname expansion will be performed too, which is going to be problematic if your parameter value contains any of *, ?, […], or [^…]. Additionally, it would be impossible to have keys or values with whitespace in them.

If you know your keys, you can define a function and pass multiple evaluations of said function:

override() {
  jq -r --arg key "$1" '.[] | select(.ParameterKey == $key) | "\(.ParameterKey)=\(env[.[1:]] // .ParameterValue)"' myfile.json
}
aws cloudformation deploy --parameter-overrides "$(override Key1)" "$(override Key2)"

Alternatively, if you are certain/can guarantee that your keys nor values will never contain line breaks or trailing blanks, the following might work:

jq -r '.[] | .ParameterValue |= (env[.[1:]] // .) | [.ParameterKey, .ParameterValue] | join("=")' myfile.json \
| xargs -L10 aws cloudformation deploy --parameter-overrides

xargs -L10 calls aws with at most 10 arguments (each line becomes one argument). Adjust to your requirements.