Search code examples
dockerkuberneteskubectl

Where does Kubernetes store cached images for use of "IfNotPresent"-CachePolicy and how to delete the cache?

I have a cluster to manage which has Pods running with Image-CachePolicy set to "IfNotPresent".

containers:
  - name: test-app
    image: myimages/test-app
    imagePullPolicy: "IfNotPresent"

I am already familiar with the difference between IfNotPresent and Always.

In our case, if the Pod gets respawned, K8s will due to "IfNotPresent" load the image from some cache instead of fetching the image from docker-hub.

What we want to achieve is to find this place where kubectl is caching the image and delete the cache in order to make K8s re-fetching the image on the next time, when the Pod gets restarted.

We don't want to apply a new Policy with imagePullPolicy set to Always because this would already fetch the latest version of the Pod, which we don't want yet. Instead, it shall only fetch new newest image on the next restart of the pod.

Is there any way to do this? Where to find this caching-location?

Solution

  • If you imagined Kubernetes was running Docker, it'd do the equivalent of:

    1. Look at docker images.
    2. If the image isn't already there, docker pull it.
    3. docker run the resulting image.

    (imagePullPolicy: Always skips the first step and always pulls the image; Never skips the pull, and will fail if the image isn't already there.)

    Each node will have its own copy of the images; where exactly depends on the specific container runtime in use. Each node in turn knows how to garbage collect images not currently in use, though old images will generally stick around until the local disk fills up (or until a tool like the cluster autoscaler deletes the node).

    A general assumption in Kubernetes is that a given registry.example.com/name/image:tag string uniquely identifies an image. Your CI/CD system should cooperate with this, and assign a distinct tag to each image build (maybe based on the commit ID or a date stamp). If you do this, you shouldn't need to adjust imagePullPolicy: from its default or try to manually remove images from nodes.