I have started working with Prometheus / Grafana. I have some data from my Kubernetes cluster that I want to dashboard. Specifically for the persistent volume claims, I want to show the current free space in a table or some such.
This metric on its own does not exist. However, I can get the total capacity using this query: px_volume_fs_capacity_bytes{}
This returns a list of values that has this structure:
px_volume_fs_capacity_bytes{app_kubernetes_io_instance="redis",app_kubernetes_io_name="redis",cluster="px-cluster-6b616c47-4d56-414c-9638-c95dcde99d4c",clusterUUID="52eff814-17e0-485e-b312-4be39d4e386f",endpoint="px-api",exported_namespace="database",instance="192.168.2.9:9001",job="portworx-service",namespace="portworx",node="kubenode02",nodeID="dfd4576e-185c-48c3-9542-8724319de0ac",pod="px-cluster-6b616c47-4d56-414c-9638-c95dcde99d4c-m988d",priority_io="high",pvc="redis-data-redis-master-0",repl="1",service="portworx-service",volumeid="777692050373571690",volumename="pvc-6de4deb8-19ac-4340-be43-2769b733dc89",app_kubernetes_io_component="master"}
There is one of these entries for each persistent volume. I can also do the same for bytes used. What I want to be able to do is something like:
for each $persistent_volume_name (px_volume_fs_used_bytes/px_volume_fs_capacity_bytes)
Note that the lable value pvc exists in both data sets so i hoped to use that as a lookup between them.
Where I would then get a table like this:
| Volume Lable | Free Space (MB) |
|---|---|
| persistent_vol_label01 | 500MB |
| persistent_vol_label02 | 100MB |
| persistent_vol_label03 | 200MB |
I could of course make a single box for each of these items but I was hoping to make one dynamic view.
Prometheus is automatically doing "foreach".
So if you guarantee that in both metrics label volumename is unique, and present in both metrics at the same time, you can simply use query
px_volume_fs_used_bytes
/ on(volumename)
px_volume_fs_capacity_bytes
It will take all px_volume_fs_used_bytes match appropriate px_volume_fs_capacity_bytes (with the same value of label volumename) and return result of division.