Search code examples
prometheusgrafanapromql

Prometheus query in Grafana with query variable

My Grafana panel query is

sum(kube_pod_container_resource_limits_cpu_cores{node=~"$workers"}) / sum(kube_node_status_allocatable_cpu_cores{node=~"$workers"})

The variable of "workers" is defined as a Prometheus query variable

label_values(kube_node_role{role="worker"}, node)

enter image description here This query will return a list of nodes that have the label of role="worker".

The question is because there might be a situation where one of the nodes doesn't contain any pod that defines CPU limits, so the metric of kube_pod_container_resource_limits_cpu_cores won't contain that host.

When using {node=~"$workers"} in the panel, the whole query will return N/A because sum(kube_pod_container_resource_limits_cpu_cores{node=~"$workers"}) return N/A.

I found out that the reason is when referring the variable $workers, it only pick up the first host, and coincidentally the first host doesn't have the metric of kube_pod_container_resource_limits_cpu_cores, so in the panel, it shows N/A. enter image description here

enter image description here

How can I fix it or improve the query to get only workers CPU limits committment? Anything wrong to my set up?

PS: I try to define a new constant variable named with hosts that has all the nodes which return by the query of label_values(kube_node_role{role="worker"}, node), but manually combine them with |(the constant variable looks like hostA|hostB|hostC). By adjusting the query with using this variable sum(kube_pod_container_resource_limits_cpu_cores{node=~"$hosts"}) / sum(kube_node_status_allocatable_cpu_cores{node=~"$hosts"}), it could display correctly! Could it might be the Grafana inside mechannism problem that when using multi values from a query variable?

Debug screenshots
The first query only contains one host. enter image description here

Solution

  • In Dashboard setting, choose the variable options as

    Hide               = empty
Multi-value        = enable
Include All option = enable

    Then select "all" from the dashboard label, so the query "node=~${workers}" could select all nodes that are filtered out by the query variable.