Search code examples
prometheusgrafana

Can I create a set of repeated rows and panels using two dependent variables?

I have the prometheus metric node_hwmon_temp_celsius which has the relevant labels chip and sensor.

For example, chip might be nvme_nvme0 or platform_coretemp_0 and sensor might be temp1, temp2, temp3, ...

I've created two variables in Grafana:

  • chip which is defined as label_values(node_hwmon_temp_celsius{job="$job"}, chip)
  • sensor which is defined as label_values(node_hwmon_temp_celsius{job="$job",chip=~"$chip"}, sensor)

As you can see, sensor depends on the value of chip. I have defined my row to repeat by chip and my panel within that row to repeat by sensor.

If I select "All" on chip then I get multiple chip rows, but the panels are repeated for every sensor across any chip, for example:

Row: chip="nvme_nvme0" contains 24 panels (but there are only 3 nvme temps): chip="nvme_nvme0", "sensor="temp1..24"}

Row chip="platform_coretemp_0" contains 24 panels (because 24 CPU cores): chip="platform_coretemp_0", "sensor="temp1..24"}

I completely understand why this is happening, it's because the sensor value is just picking up all the possible values on a dashboard level for any chip value.

My desired behaviour is:

Row: chip="nvme_nvme0" contains 3 panels: chip="nvme_nvme0", "sensor="temp1..3"}

Row chip="platform_coretemp_0" contains 24 panels: chip="platform_coretemp_0", "sensor="temp1..24"}

That is, I want to only repeat panels for the chip/sensor pairs that have data.

Solution

  • It is impossible to do so.

    Two easily imaginable way to achieve something like that would be:

    1. Value of sensor withing row recalculated for corresponding chip.
      Nothing of the similar manner is implemented, and I doubt that ever be: rather convoluted logic of variable dependencies will cause more problems, then it solves.

    2. Conditional hiding of panel.
      There were a couple questions about hiding panel based on value or query conditions here and at Grafana's communities, but AFAIK, they are unresolved as of now.

    In my opinion best course of action, available at this point, is to use single dimensional repeat: Use repeat panel over chip and show every sensor on this single panel. This way you'll have built-in hiding on non-existent chip-sensor pairs.