Search code examples
prometheusgrafanametricspromql

Prometheus: how to get label_value() in query

So I have 2 metrics, I need to get label values from first metric and then query over 2nd metric, where {param="label_values_from_1st_metric"}. For ex:

metric_1{instance="abc"} - returns 3 timeseries:

metric_1{name="my-service", env="production", host="example-0.org"}

metric_1{name="my-service", env="production", host="example-1.org"}

metric_1{name="my-service", env="production", host="example-2.org"}

Next I need to make query over 2nd metric using host values from 1st query.

metric_2{domain="exmple-0,1,2.org"}

So the question is, how I can pass label_values to 2nd query? As far as I understand, I can use label_values() only for variables in grafana panel, so I can't write one query that will do that for me.

Solution

  • Okay, finally, I've got it. So As Pulak Kanti suggested, I used group_left. However it returned error: many-to-many mathcing is not allowed. So what I've done:

    1. Labels between metric_1 and metric_2 do not match, however it is pretty simple to fix it, we need to add label_replace() func label_replace(metric_2,"host","$1","domain", "(.+)"). This expression adds to metric_2 label host with values copied from label domain. From our point of view we've just renamed label domain to label host. And this is great news, because now metric_1 and metric_2 have one common label - host. So the query from this step would look something like this:

      label_replace(metric_2,"host","$1","domain", "(.+)") + on (host) group_left(domain) metric_1{instance="abc"}

    2. Now our query at least doesn't retrun any error, however, it returns unexpected result-this is because it multiplies values from metric_2 with values from metric_1. Because we need only values from metric_2, we can just null metric_1, so it ends up being: label_replace(metric_2,"host","$1","domain", "(.+)") + on (host) group_left(domain) 0*metric_1{instance="abc"}

    I do understand that this query isn't effective, however it worked for me. I would be very grateful if someone would help and make this query more efficient. BTW: here it is not important what is inside group_left() brackets, because it influences only what label we displays, for me it is not important, so label_replace(metric_2,"host","$1","domain", "(.+)") + on (host) group_left() 0*metric_1{instance="abc"} also works.