Search code examples
prometheusgo-templates

How to use the {{ $labels.instance }} syntax within the Prometheus query

Given the Prometheus query syntax below for a annotation field "currentNumber" in the alert.yaml

currentNumber: "{{ with query \"max_over_time(json_exporter_resultList__0::currentDeviceNumber[1h])\" -}}
    {{- . | first | value -}}
    {{- end }}"

It works but might query an incorrect value if there are metrics from multiple instances.

So, I have to query by the label as syntax below which works perfectly on Prometheus web console.

max_over_time(json_exporter_resultList__0::currentDeviceNumber{instance='test'}[1h])

However, I couldn't get it right to be used within the "with query" syntax with the {{ $label.instance }} syntax.

currentNumber: "{{ with query \"max_over_time(json_exporter_resultList__0::currentDeviceNumber{instance='{{ $label.instance }}'}[1h])\" -}}
    {{- . | first | value -}}
    {{- end }}"

Either the queried value is empty or syntax is error by escaping the ' or changing to use ` etc..

Solution

  • According to examples in Prometheus' documentation you should use printf to prepare query if it uses labels of your current alert, and feed result into query using pipe.

    In your example this should result in something like this:

    currentNumber: "{{ with printf "max_over_time(json_exporter_resultList__0::currentDeviceNumber{instance='%s'}[1h])" .Labels.instance | query  -}}
    {{- . | first | value -}}
{{- end }}"