Context: Prometheus and Grafana running on a Kubernetes cluster.
Is there a way to display results from multiple related queries in one view (a table)? To explain further, the queries are very similar, they just filter on a set of possible label values available in a metric. The result data is identical in form for each query.
Here is some concrete information.
The query:
(kube_pod_status_phase{phase=~"$pod_phase"} != 0) + on(pod) group_left(node) (0 * kube_pod_info{node=~"$node"})
The whole on/group_left logic is allowing me to associate a node name with the data from kube_pod_status_phase, since that is not available in that metric. The important stuff is what's coming from kube_pod_status_phase.
What I'd like to do is run the query for each node to get and display the number of pods that are in each of the valid states (Running, Pending, etc.). I would think the query would just then have a count function applied.
Example:
| Node | Running | Failed | Pending | ... |
|---|---|---|---|---|
| node-1 | 10 | 1 | 0 | ... |
| node-2 | 12 | 2 | 1 | ... |
| ... | ... | ... | ... | ... |
I've poked around in the Grafana documentation, especially in the transformations, but can't find a way to make this work.
UPDATE:
After a little research, I set up the multiple queries and then tried using a merge transformation (https://grafana.com/docs/grafana/latest/panels-visualizations/query-transform-data/transform-data/#merge). I get an error indicating: Merge has no effect when applied on a single frame. I do not know what that means or how to correct the problem.
You need a query like this:
sum by (node, phase) (
kube_pod_status_phase + on(pod) group_left(node) (0 * kube_pod_info{node=~"$node"})
)
It calculates number of pods in each status per node.
Set query options (collapsible options pane is under the query) to:
After that, add Transformation Grouping to matrix, with the following options:
In result you'll get table similar to described in question, with rows presenting your nodes, columns - phases of pods, and cell values equal to number of pods on respective node in respective state.