Probes not found on $PATH

Question

I am redeploying a K3s deployment from a few months ago. Then, it worked perfectly, with no problems. However, when I try deploying it now - after making some other fixes, I get the following error:

Warning  Unhealthy  32m                   kubelet            Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "8078b7c54b9bb1609451ae1c2e832ede0670f264490f6ee34e334673fd025681": OCI runtime exec failed: exec failed: unable to start container process: exec: "grpc_health_probe": executable file not found in $PATH: unknown

This is that the .yaml file I am using for the deployment.

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vei-deployment
spec:
  replicas: 1
  selector:
    matchLabels:
      app: server-pod
  template:
    metadata:
      labels:
        app: server-pod
    spec:
      containers:
        - name: server-pod
          image: myname/mydeployment:latest
          env:
          - name: AWS_ACCESS_KEY_ID
            value: $AWS_ACCESS_KEY_ID
          - name: AWS_SECRET_ACCESS_KEY
            value: $AWS_SECRET_ACCESS_KEY
          ports:
            - name: grpc
              containerPort: 50051
          livenessProbe:
            exec:
              command:
                - grpcurl
                - -plaintext
                - localhost:50051
                - ping.Pinger/Ping
          readinessProbe:
            exec:
              command:
                - grpc_health_probe
                - -addr=:50051

The same error is thrown for every command in both the liveliness probe and the readiness probe. Has something changed with respect to probes in K3s over the past few months?

Answer 1

Warning Unhealthy 32m kubelet Readiness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "8078b7c54b9bb1609451ae1c2e832ede0670f264490f6ee34e334673fd025681": OCI runtime exec failed: exec failed: unable to start container process: exec: "grpc_health_probe": executable file not found in $PATH: unknown

This error could be caused by a few different scenarios as addressed below :

1. It could be that the container image you are using is missing the grpchealthprobe executable, or it could be that the image is not correctly configured to find the executable in the $PATH. If the image is correctly configured, it could also be that the kubelet is not able to access the container image.

2. As @DazWilkin It looks like the issue is that the grpchealthprobe binary is not present in your Kubernetes cluster.It looks like your Kubernetes deployment is failing due to a missing executable file. This error can occur when the file needed by the readiness probe is not present in the container. To solve this, you will need to make sure that the file is present in the container. This can be done by adding the file to the container image or by mounting the file into the container.you'll need to make sure that the "grpchealthprobe" executable is in the $PATH environment variable, or specify a full path to the executable in the readiness probe configuration. Additionally, you may need to ensure that the permissions of the executable are set correctly so that it can be executed. Once the file is present, the readiness probe should start working properly. you should be able to deploy your K3s deployment without any further issues.

For more info follow this doc.