Secrets Manager Update Secret - Secret String additional JSON Encoding

Question

I am using the AWS CLI to update a JSON secret containing x509 certificates. The payload that I am passing into the --secret-string parameter is the encoding I am expecting, but secretsmanager is performing additional encoding of the newline characters when it is uploaded and stored in AWS.

Problem

The problem I am getting is the PEM and Key contents are not entirely being unescaped when unmarshalling the secret. I have an application server in Go that when passing the secrets into tls.X509KeyPair() it is unable to recognize the contents of the certificates.

I can paste the payload from the CLI command into the secret manually on the web and it retains the existing formatting and my code works fine.

I am primarily looking to see if there is something I can do to fix the upload encoding instead of having to format / unescape the PEM/Key contents in Go when fetching the secret.

Example:

CLI Command:

aws secretsmanager update-secret --secret-id my-secret --secret-string '{"PEM":"-----BEGIN CERTIFICATE-----\n...\n...\n...\n-----END CERTIFICATE-----\n","KEY":"..."}'

Actual Uploaded Secret:

{"PEM":"-----BEGIN CERTIFICATE-----\\\\n...\\\\n...\\\\n...\\\\n-----END CERTIFICATE-----\\\\n",,"KEY":"..."}

I am using awk to join the PEM contents and jq for updating + stringifying the JSON. The stringified JSON is the correct format I am looking for.

awk: STRING_PEM=$(awk '{printf "%s\\n", $0}' $PemFile)

jq: SECRET_STRING=$(jq -r 'tostring' $UPDATED_SECRET)

Answer 1

I can confirm that a secret set by this command

aws secretsmanager update-secret --secret-id my-secret --secret-string '{"PEM":"-----BEGIN CERTIFICATE-----\n...\n...\n...\n-----END CERTIFICATE-----\n","KEY":"..."}'

works as expected (see the Go demo at the bottom of this answer).

And here is a simplified command that does the whole thing:

aws secretsmanager update-secret --secret-id -secret --secret-string "$(jq -n --arg cert "$(cat /path/to/cert/file)" --arg key "$(cat /path/to/key/file)" -c '{"PEM": $cert, "KEY": $key}')"

Here is a Go demo used to verify the uploaded content:

package main

import (
    "context"
    "crypto/tls"
    "encoding/json"

    "github.com/aws/aws-sdk-go-v2/aws"
    "github.com/aws/aws-sdk-go-v2/config"
    "github.com/aws/aws-sdk-go-v2/service/secretsmanager"
)

func main() {
    secretName := "test1"
    region := "ap-southeast-1"

    config, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion(region))
    if err != nil {
        panic(err)
    }

    svc := secretsmanager.NewFromConfig(config)

    input := &secretsmanager.GetSecretValueInput{
        SecretId: aws.String(secretName),
    }

    result, err := svc.GetSecretValue(context.TODO(), input)
    if err != nil {
        panic(err)
    }

    var secret struct {
        PEM string `json:"PEM"`
        KEY string `json:"KEY"`
    }
    {
    }

    if err := json.Unmarshal([]byte(*result.SecretString), &secret); err != nil {
        panic(err)
    }

    if _, err := tls.X509KeyPair([]byte(secret.PEM), []byte(secret.KEY)); err != nil {
        panic(err)
    }
}