Prometheus Docker fails to start with `Template: (dynamic): parse: template: :10: undefined variable "$labels"`

Question

I am trying to run Prometheus on Nomad. Everything things look fine but when I add an alert rule with templating in annotations it fails.

here is a simple alert rule:

groups:
- name: alertmanager.rules
  rules:
  - alert: AlertmanagerDown
    expr: up{job="alertmanager"} == 0
    for: 1s
    labels:
      severity: critical
    annotations:
      description: 'Alertmanager on {{ $labels.role }} instance {{ $labels.node }} has not produced any metrics for 5 minutes'
      summary: 'Alertmanager is down on {{ $labels.role }} instance {{ $labels.node }}'

Container fails with Template: (dynamic): parse: template: :10: undefined variable "$labels"

If I remove annotations section it starts without any issue.

Answer 1

Had to set left_delimiter and right_delimiter to something other than {{ and }} https://www.nomadproject.io/docs/job-specification/template.html#left_delimiter