Search code examples
kuberneteskubernetes-secrets

Mounting secrets directly to /run/secrets in Kubernetes

I do have an existing application that used docker and docker compose so far. I want to operate this app in a Kubernetes cluster. Shouldn't be a big deal, right? But so far I failed because of the used secrets.

The application expects a secret to be present in a file at: /run/secrets/webhook_secret, where webhook_secret is the file containing the secret.

I created a secret with kubectl like this:

   kubectl create secret                                                       
      generic webhook-secret \
      --from-literal=webhook_secret=123 \
      --namespace my-app

I tried to mount the secret in the manifest with...

  ...
  volumeMounts:
  - name: secrets
    mountPath: "/run/secrets"
    readOnly: true
  volumes:
  - name: secrets
    secret:
      secretName: webhook-secret
  ...

But then the pod is not able to start, as Kubernetes also tries to mount the same directory and so issues with run/secrets/kubernetes.io occur...

Warning Failed 3m1s (x4 over 3m37s) kubelet Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/e79f634a-2abe-4c47-ae50-c4beb5b66ae6/volumes/kubernetes.io~projected/kube-api-access-grl42" to rootfs at "/var/run/secrets/kubernetes.io/serviceaccount": mkdir /run/containerd/io.containerd.runtime.v2.task/k8s.io/my-app-container/rootfs/run/secrets/kubernetes.io: read-only file system: unknown

Any idea how I can provide the secret as required by the application?

Regards and thanks, Thomas

Solution

  • It might be perfectly possible to mount your own secrets into the /var/run/secrets or /run/secrets path. Even in the scenario where Kuberbetes is concurrently using that path for system-related tasks. This commonly happens when, for instance, you are running a cluster with 2+ nodes and Kubernetes starts projecting it's system objects like secrets, tokens, certifates, etc. into /var/run/secrets/kubernetes.io/serviceaccount/.

    I don't know the exact specifics of your case, but you might want to look into subPath. When your application expects secrets to be available in /run/secrets, you can do so something like:

    ...
containers:
    - name: myContainer
      image: "someImage:1.0.0"
volumeMounts:
  - name: mySecretVolumeName
    mountPath: "/run/secrets/webhook_secret"
    readOnly: true
    subPath: webhook-secret
volumes:
  - name: mySecretVolumeName
    secret:
      secretName: secrets
      items:
        - key: webhook-secret
          path: webhook-secret
...

    People started bumping into the issue you're experiencing (if I understand you correctly) a few years ago: https://github.com/kubernetes/kubernetes/issues/65835

    Hope this helps you out!