Search code examples
kubernetesprometheusgrafana

How to transform results from related metrics into a single state value


I am using Prometheus and Grafana to collect and display pod/container status for a Kubernetes cluster. I'm collecting the information from the following metrics:

kube_pod_container_status_running
kube_pod_container_status_terminated
kube_pod_container_status_waiting

Note: I left a fourth metric, kube_pod_container_status_ready out as it seems to be a duplicate of kube_pod_container_status_running. If I am mistaken, please let me know what the difference is.

Each metric returns a 0 or 1 result, where 1 indicates the container is currently in that state (e.g. runnning). I'm making the assumption that at any given time, only one of these metrics should have a value of 1 for a given set of labels representing a specific container in the cluster. From what I've seen, each metric collects the same set of label dimensions.

What I would like to do is display a table of container information of interest (pod, container, namespace, etc.) plus a column indicating the current state (Running, Terminated, etc.). I may need to include other queries to integrate other information not available from this current set.

I have tried a couple of experiments that have allowed me to collect the information into a single table view, but cannot figure out how to translate the 3 metric results into a single state representation. So, for example: [running=1, terminated=0, waiting=0] into "Running", or [running=0, terminated=0, waiting=1] into "Waiting".

Any help on this would be appreciated.


Solution

  • You metrics seems rather strange, and even against Prometheus' recommendations. Usually such metrics would be in format kube_pod_container_status{status="waiting"}.

    You can convert your metrics to conventional format, and display them in table.

    Use query

    label_replace(
     label_replace(
      {__name__=~"kube_pod_container_status_.*"} == 1
      ,"status","$1","__name__","kube_pod_container_status_(.*)")
    ,"__name__","kube_pod_container_status","__name__",".*")
    

    And switch format of you query (under your query editor) to "Table".

    If you'll use panel "Table", you'll see metrics like you described.