Search code examples
kuberneteskubernetes-pod

error killing pod: failed to "KillPodSandbox" for device or resource busy


When I want to restart the kubernetes(v1.21.2) statefulset pod, the pod are stuck with terminating status, and the log shows like this:

error killing pod: failed to "KillPodSandbox" for "8aafe99f-53c1-4bec-8cb8-abd09af1448f" with KillPodSandboxError: "rpc error: code = Unknown desc = failed to check network namespace closed: remove netns: unlinkat /var/run/netns/cni-f9ccb1de-ed43-dff6-1b86-1260e07178e6: device or resource busy"

the pod terminate for hours but still stuck. why did this happen? what should I do to fixed this problem?


Solution

  • I think force deletion can be a workaround for this issue.

    In order to delete the affected pod that is in the terminating state, please refer to the documentation. In case the pod still does not get deleted then you can do the force deletion by following documentation.

    Please note that when you force delete a StatefulSet pod, you are asserting that the Pod in question will never again make contact with other Pods in the StatefulSet and its name can be safely freed up for a replacement to be created.

    You can also try these workarounds to quickly mitigate this

    1. Run the command below to remove all pods in the terminating state.

    for p in $(kubectl get pods | grep Terminating | awk '{print $1}'); do kubectl delete pod $p --grace-period=0 --force;done

    2.Set finalizer value in the deployment YAML to null.