Search code examples
kubernetessshssh-keyskubernetes-secrets

'Read-only file system' error when using kubernetes secret for ssh auth

I am trying to establish SSH authentication between Jenkins and GitHub. For the same, I am using the kubernetes secret to store the private and public key and I am mounting the secrets when the pod is created. Command I have used to create the secret:

kubectl create secret generic test-ssh --from-file=id_rsa=id_rsa --from-file=id_rsa.pub=id_rsa.pub --namespace jenkins

and mapped it in pod configuration as:

volumes:
  - secretVolume:
      mountPath: "/root/.ssh"
      secretName: "test-ssh"

When the pod is created, I can see that the secret is mapped correctly in the ~/.ssh folder as shown below.

enter image description here

but the problem is the ~/.ssh folder itself has the sticky bit permission enabled

enter image description here

and this is preventing the builds adding the known_hosts file when ssh-keyscan command is executed

ssh-keyscan github.com >> ~/.ssh/known_hosts
bash: ~/.ssh/known_hosts: Read-only file system

I was hoping to achieve one of the two solutions I can think of

  1. Remove the sticky permissions from ~/.ssh folder after it is created
  2. While mounting the kubernetes secret, mount it without sticky permissions

Could anyone help me to understand if there is a possibility to achieve this? I have already tried chmod -t .ssh and it gives me the same error chmod: changing permissions of '.ssh': Read-only file system

The owner of the ~/.ssh folder is root and I have logged in as root user. I have confirmed this by running the whoami command.

Solution

  • Secrets aren't mounts that allow for permissions to be alter, so you have to do some trickery to get it to work. Recently I ran into a similar issue, and had to make it work. We use a custom image to perform the ssh calls, so we added the .ssh dir and the known_hosts files to the image; setting the permissions in the Dockerfile. I then used the subPath to target the id_rsa and id_rsa.pub files in the deployment. I don't have my mappings handy but can tomorrow if necessary to show what I mean. I think the 'official' answer was to mount the files in a different location and cp them where they needed to be in an initContainer using a shared mount or in the entrypoint file/binary.