ssh keys inside cloud build

Question

I am trying to create a service that would run in a container, connect to a remote host by ssh and perform several commands remotely. This service will be built and run in cloud build. Right now I am stuck with trying to inject SSH keys into a container - build fails with the error:

Step #2 - "run replication command": Load key "/root/.ssh/id_rsa": invalid format

My keys were created with ssh-keygen so they are probably correct. There seems to be some sort of issue with SSH keys, but I have no idea what is wrong. What I have already tried:

1. As this is the follow-up to this issue, I have already tried injecting the keys through the secret manager at build stage. This failed because builder refuses to perform substitutions properly and passes the variable name instead of the value (that is, I have a secret env variable called PRIVATE, and when I reference it as $$PRIVATE, builder sets value to $PRIVATE which is not what I need);
2. I have tried creating the secrets in different ways:
   • echo $(cat container_private_key.pem) | gcloud secrets create private-key --data-file=-
   • echo -n <secret value here> | gcloud secrets create private-key --data-file=-
   • gcloud secrets create private-key --data-file=container_private_key.pem
     None of these work, cloud build still complains about invalid format. I tried adding/removing a newline at the end of the file, it did not help.
3. I also tried inserting keys both with and without headers, that had no positive effect.

Right now the keys are injected as follows:

mkdir -p /root/.ssh && chmod 0700 /root/.ssh && echo $CONTAINER_PRIVATE_KEY > /root/.ssh/id_rsa && \
    echo $CONTAINER_PUBLIC_KEY > /root/.ssh/id_rsa.pub && chmod 400 /root/.ssh/id_rsa && \
    chmod 600 /root/.ssh/id_rsa.pub

As you can see, the values are taken from the environment variables, and these variable actually do contain the keys, I tried echoing them. I assume that the issue is somehow related to the keys being mangled at some stage between my computer, secret manager and echoing the values from environment variables, but I cannot understand where exactly it might happen.

Answer 1

I did some digging and found out that my hypothesis about SSH keys getting mangled at some point was correct. The problem illustrated:

1. I submit a secret to Secret Manager in a file (no problem here, the file seems to upload in its original form);
2. My secret gets referenced and exported as an environment variable to Cloud Build and that is where things get bad. Somewhere around this place the key gets stripped of all the special unprintable symbols and SSH will consider it a malformed key in the next step;
3. The value enters the file successfully, but since it is malformed, SSH does not care and will not use the key.

So, to work around this issue I did the following: I encoded my keys into base64 on my host machine and decoded them at the very last moment. Note the calls to base64

mkdir -p /root/.ssh && chmod 0700 /root/.ssh && echo "$CONTAINER_PRIVATE_KEY" | base64 --decode > /root/.ssh/id_rsa && \
    echo "$CONTAINER_PUBLIC_KEY" | base64 --decode > /root/.ssh/id_rsa.pub && chmod 400 /root/.ssh/id_rsa && \
    chmod 600 /root/.ssh/id_rsa.pub