Search code examples
prometheusmetricsobservability

how to create queries in prometheus with wildcard


my application emits counter metrics something like this:

app_counters_some_client_response_code_201_count_value{}
app_counters_some_client_response_code_400_count_value{}
app_counters_some_client_response_code_404_count_value{}

I want to create a graphana dashboard which can show all these different metric values in the same dashboard with the status codes as the legend.. how can I write such a single query using a wildcard as there can be multiple response codes and we don't have the complete list of possible values, so we won't be able to write a different query for each metric


Solution

  • First of all, such metrics are a bad idea for exactly the problem you are having. Please read recommendations on exposing metrics, and consider changing your metrics to use labels. Something like

    yourApp_yourScope_responses_total{code="201"}
    yourApp_yourScope_responses_total{code="400"}
    yourApp_yourScope_responses_total{code="404"}
    

    That being said, you could use a query:

    label_replace({__name__=~"app_counters_some_client_response_code_\d+_count_value"}, "http_code", "$1", "__name__", "app_counters_some_client_response_code_(\d+)_count_value")
    

    and specify legend in the Grafana panel as {{http_code}}

    Explanation:
    This query:

    1. selector {__name__=~"app_counters_some_client_response_code_\d+_count_value"} gets all time series with name matching regex app_counters_some_client_response_code_\d+_count_value
    2. Then label_replace extracts digits from the names of the metrics and puts them into the label http_code.