Search code examples
prometheuspromql

Querying prometheus label values with metric values

Is there a way to get the label values as the Value of a prometheus query. For example, i need to filter out the instances which has more than 80 % CPU usage. I need the list of ip addresses as the output. Is it possible or will the Value column will only be the metric values.

Expected output:

Value
10.21.123.21
72.34.128.31
Solution

  • Using HTTP API

    You can get the instances with more than 80% of CPU usage running the following command:

    curl --silent --globoff --request GET "http://PROMETHEUS-SERVER/api/v1/query?query=100-100*avg%20by%20(instance)(irate(node_cpu_seconds_total{mode=\"idle\"}[5m]))>80" | jq --raw-output '.data.result[].metric.instance'

    Using Grafana

    If you're using Grafana you can add a variable at "Dashboard settings" > "Variables" > "New"

    Query Options:

    Data source: Prometheus

Query: query_result(100 - 100 * avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) > 80)

Regex: /.*\"(.*)\".*/