Search code examples
prometheusgrafanametricspromql

Aggregate different prometheus metrics in same graph


I have an application exposing different metrics to prometheus. This applications was very specific at the beginning, and so, it was exposing highly specific metrics like service_name_especific_event_processed{event_type="(type1|type2)", result="(acked|discarded)"}, but then we made it more general purpose in our ecosystem, and we started to expose more general metrics like service_name_event_processed{event_type="one_of_a_range_of_types", result="(acked|discarded)"}.

As you can see, even though they are different metrics, they share the same set of labels [event_type, result] and we would like to create a panel in Grafana that shows both of them aggregated as if they were the very same metric, just grouping them by event_type and result, so we would have a dropdown selector filled with all different event types that would allow us to show the metrics for a specific event type. That said, our panel for just one of the metrics would be populated by a promql query like sum(rate(service_name_event_processed{event_type=~"$event_type"}[1m])) by (result). And to have both metrics being aggregated and shown in the same panel, and being able to split by event_type, the query that comes to my mind is something like sum(rate(service_name_specific_event_processed{event_type=~"$event_type"}[1d])) by (result) + sum(rate(service_name_event_processed{event_type=~"$event_type"}[1d])) by (result). That query makes the trick, but it has a problem, if there aren't values for any of the metric, it returns an empty result and so it shows nothing in the graph.

That being said, do you know any alternative way of achieving the result I'm looking for? (of course I know I could change the name of the specific metric have just one metric differentiated by its event_type and result labels, but that's not an option for now)


Solution

  • As @valyala suggested in the comments, to achieve what I was looking for, I just had to replace + by or operator. So the resulting query is as follows

    sum(rate(specific_event_processed{event_type=~"$event_type"}[1d])) by (result) or sum(rate(event_processed{event_type=~"$event_type"}[1d])) by (result)