Search code examples
kuberneteskubelet

What does Kubelet use to determine the ephemeral-storage capacity of the node?

I have Kubernetes cluster running on a VM. A truncated overview of the mounts is:

$ df -h
Filesystem                    Size  Used Avail Use% Mounted on
/dev/sda1                      20G  4.5G   15G  24% /
/dev/mapper/vg001-lv--docker  140G   33G  108G  23% /var/lib/docker

As you can see, I added an extra disk to store the docker images and its volumes. However, when querying the node's capacity, the following is returned

Capacity:
  cpu:                12
  ephemeral-storage:  20145724Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             65831264Ki
  nvidia.com/gpu:     1
  pods:               110

ephemeral-storage is 20145724Ki which is 20G, referring to the disk mounted at /.

How does Kubelet calculate its ephemeral-storage? Is it simply looking at the disk space available at /? Or is it looking at another folder like /var/log/containers?

This is a similar post where the user eventually succumbed to increasing the disk mounted at /.

Solution

  • Some theory

    By default Capacity and Allocatable for ephemeral-storage in standard kubernetes environment is sourced from filesystem (mounted to /var/lib/kubelet). This is the default location for kubelet directory.

    The kubelet supports the following filesystem partitions:

    1. nodefs: The node's main filesystem, used for local disk volumes, emptyDir, log storage, and more. For example, nodefs contains /var/lib/kubelet/.
    2. imagefs: An optional filesystem that container runtimes use to store container images and container writable layers.

    Kubelet auto-discovers these filesystems and ignores other filesystems. Kubelet does not support other configurations.

    From Kubernetes website about volumes:

    The storage media (such as Disk or SSD) of an emptyDir volume is determined by the medium of the filesystem holding the kubelet root dir (typically /var/lib/kubelet).

    Location for kubelet directory can be configured by providing:

    1. Command line parameter during kubelet initialization

    --root-dir string Default: /var/lib/kubelet

    1. Via kubeadm with config file (e.g.)
    apiVersion: kubeadm.k8s.io/v1beta3
kind: InitConfiguration
nodeRegistration:
  kubeletExtraArgs:
    root-dir: "/data/var/lib/kubelet"

    Customizing kubelet:

    To customize the kubelet you can add a KubeletConfiguration next to the ClusterConfiguration or InitConfiguration separated by --- within the same configuration file. This file can then be passed to kubeadm init.

    When bootstrapping kubernetes cluster using kubeadm, Capacity reported by kubectl get node is equal to the disk capacity mounted into /var/lib/kubelet

    However Allocatable will be reported as: Allocatable = Capacity - 10% nodefs using the standard kubeadm configuration, since the kubelet has the following default hard eviction thresholds:

    • nodefs.available<10%

    It can be configured during kubelet initialization with: -eviction-hard mapStringString Default: imagefs.available<15%,memory.available<100Mi,nodefs.available<10%

    Example

    I set up a test environment for Kubernetes with a master node and two worker nodes (worker-1 and worker-2).

    Both worker nodes have volumes of the same capacity: 50Gb.

    Additionally, I mounted a second volume with a capacity of 20Gb for the Worker-1 node at the path /var/lib/kubelet. Then I created a cluster with kubeadm.

    Result

    From worker-1 node:

    skorkin@worker-1:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        49G  2.8G   46G   6% /
...
/dev/sdb         20G   45M   20G   1% /var/lib/kubelet

    and

    Capacity:
  cpu:                2
  ephemeral-storage:  20511312Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             4027428Ki
  pods:               110

    Size of ephemeral-storage is the same as volume mounted at /var/lib/kubelet.

    From worker-2 node:

    skorkin@worker-2:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        49G  2.7G   46G   6% /

    and

    Capacity:
  cpu:                2
  ephemeral-storage:  50633164Ki
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             4027420Ki
  pods:               110