Search code examples
kubernetesprometheuspromql

Track Kubernetes Deployments with kube_pod_container_info in Prometheus

I am looking for a query to see when a container's image tag changes within a pod with Prometheus. I have tried a few different ways of doing and I think I am pretty close but the issue i can't seem to solve is how to compare the current record with the record from 5 minutes ago and only get results when the image changes but still include the pod and container data. I have this query so far kube_pod_container_info{namespace="test"} != on(image) kube_pod_container_info{namespace="test"} offset 5m

I get an error with this query because there are multiple containers running the same image. So ideally i would match the current records/past records on pod/container and then check to see if the images didn't match. here is an example of some of the data: current data

kube_pod_container_info{namespace="test",pod="test-pod", container="test-container-1", image="test-image:v1"}
kube_pod_container_info{namespace="test",pod="test-pod", container="test-container-2", image="test-image:v2"}

offset 5m data:

kube_pod_container_info{namespace="test",pod="test-pod", container="test-container-1", image="test-image:v1"}
kube_pod_container_info{namespace="test",pod="test-pod", container="test-container-2", image="test-image:v1"}

From the data above would want to query to return kube_pod_container_info{pod="test-pod", container="test-container-2", image="test-image:v2"} as its container image was recently changed to the tag v2. Let me know if you need anymore clarification. Thanks in advance for the help!

Solution

  • You can use something like this:

    kube_pod_container_info{namespace="test"} == ignoring(image)
    kube_pod_container_info{namespace="test"} offset 5m 
  unless ignoring(image)
    kube_pod_container_info{namespace="test"} ==
      kube_pod_container_info{namespace="test"} offset 5m

    Here we check if kube_pod_container_info matches all labels except image wtih itself 5 min ago, but not matches them all, then it means that image changed.

    EDIT: Query to save all labels suggested by @JeffW

    (kube_pod_container_info{namespace="test"} == ignoring(image)
    kube_pod_container_info{namespace="test"} offset 5m 
  unless ignoring(image)
    kube_pod_container_info{namespace="test"} ==
      kube_pod_container_info{namespace="test"} offset 5m )
* on(pod,cotainer) group_left(image) kube_pod_container_info{namespace="test"}