How to fix disk-pressure on k8s node?

Question

K8s 1.24.17 Containerd as a container runtime(instead of docker) How to fix disk-pressure taint on node? I tried:

sudo crictl rmi --prune && sudo crictl rm -a

but still nothing happened, only a little bit of space emptied. Then i tried:

kubectl taint node foo-node1 node.kubernetes.io/disk-pressure-

After several seconds running kubectl describe node foo-node1 but my node is still has disk-pressure taint

How to fix it?

Result of df -h

Answer 1

If crictl command doesn’t resolve your issue try the below steps to fix disk pressure taint issue:

First, Detect node pressure by running $ kubectl get nodes command , Any nodes whose status includes DiskPressure=True are under disk pressure .

Run the command $kubectl describe pod <pod-name\> to get more information about each of the Pods running on the node.

In the Volume section, check which PVCs (if any) the Pod is using & then check storage resources mapped to the PVC to figure out which data actually exists in them.

Verify if log files are excessively large, which can trigger disk pressure because K8s is not supporting log rotation and they can grow to hundreds of Gigs.

Use lsof to show the list of used files and sort them at https://unix.stackexchange.com/a/382696/380398

In addition to the disk space consumed by individual Pods, Kubernetes components can also consume additional storage space on a node.

The exact storage paths vary between distributions and container runtimes. Kubernetes typically uses directory /var and in subdirectories like /var/lib and /var/lib/containers locations to store data like container images, which could also be sucking up disk space and causing node disk pressure issues.

Adding storage capacity to a node is also the one way to resolve node disk pressure.

After cleanup try running the below command to remove the taint for disk pressure.

$ kubectl taint nodes <nodename> node.kubernetes.io/disk-pressure-