I'm trying to aggregate some promql queries for my Grafana dashboard that should give me some information about the persistent volume claims within my Kubernetes cluster. The problem is, that these queries don't have a unique identifier (the combination of the labels cluster, namespace, persistentvolumeclaim) can be used for this.
A simplified example of what I'm dealing with, I've got 3 separate queries:
sum by (cluster, namespace, persistentvolumeclaim) (kubelet_volume_stats_capacity_bytes{}/1024/1024/1024)
sum by (cluster, namespace, persistentvolumeclaim) (kubelet_volume_stats_used_bytes{}/1024/1024/1024)
sum by (cluster, namespace, persistentvolumeclaim) (kubelet_volume_stats_used_bytes{}/kubelet_volume_stats_capacity_bytes{} * 100)
Within Grafana I am only able to transform these queries via an outer join, which only enables me to select 1 label as distinguisher. These obviously would not work for me, as values should only be combined if these 3 labels are unique. What would be the correct way to aggregate this information?
PromQL provides a function for joining multiple labels into a single label - label_join. For example, the following query joins cluster, namespace and persistentvolumeclaim labels into a single one - combined_label with the ; delimiter between original pabel values:
label_join(..., "combined_label", ";", "cluster", "namespace", "persistentvolumeclaim")
Where ... must be substituted by one of the queries above. Then query results can be joined in Grafana by the combined_label.