Why helm template function is not resolveing $labels var?

I am defining a PrometheusRule as follow:

prometheusRule:
  rules:
    - alert: SSLCertExpiringSoon
      expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 10
      for: 0m
      labels:
        severity: warning
      annotations:
        summary: Blackbox SSL certificate will expire soon (instance {{ $labels.instance }})
        description: "SSL certificate expires in 30 days\n  VALUE = {{ $value }}\n  LABELS = {{ $labels }}"

And the template yml from helm chart:

{{- if .Values.prometheusRule.enabled }}
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: {{ template "prometheus-blackbox-exporter.fullname" . }}
  {{- with .Values.prometheusRule.namespace }}
  namespace: {{ . }}
  {{- end }}
  labels:
    {{- include "prometheus-blackbox-exporter.labels" . | nindent 4 }}
    {{- with .Values.prometheusRule.additionalLabels -}}
{{- toYaml . | nindent 4 -}}
    {{- end }}
spec:
  {{- with .Values.prometheusRule.rules }}
  groups:
    - name: {{ template "prometheus-blackbox-exporter.name" $ }}
      rules: {{ tpl (toYaml .) $ | nindent 8 }}
  {{- end }}
{{- end }}

when I run helm template, tpl func it is not resolving the $labels and $values vars. When I remove annotations then the helm template is not complaining anymore. Where do fail?

error:

Error: template: prometheus-blackbox-exporter/templates/prometheusrule.yaml:18:16: executing "prometheus-blackbox-exporter/templates/prometheusrule.yaml" at <tpl (toYaml .) $>: error calling tpl: error during tpl function execution for "- alert: SSLCertExpiringSoon\n  annotations:\n    summary: Blackbox SSL certificate will expire soon (instance {{ $labels.instance\n      }})\n  expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 10\n  for: 0m\n  labels:\n    release: prometheus\n    severity: warning\n- alert: SSLCertExpiringSoon\n  annotations: null\n  expr: probe_ssl_earliest_cert_expiry - time() < 86400 * 3\n  for: 0m\n  labels:\n    severity: critical": parse error at (prometheus-blackbox-exporter/templates/prometheusrule.yaml:3): undefined variable "$labels"

Solution

Prometheus's alerting rules also use {{ ... $variable ... }} syntax, similar to Helm but with a different variant on the Go text/template syntax. When you pass this file through tpl, Helm tries to evaluate the embedded {{ ... }} template and evaluate any blocks there. Since $labels and $value aren't local variables defined at the Helm level, you get this error.

If you just want Prometheus to see this file as-is, and you don't need to replace anything at the Helm level (the file doesn't include references to .Values) then you don't need tpl

      rules: {{ toYaml . | nindent 8 }}

If you do need tpl, then inside the included file you need to cause {{ to be emitted as a string and not processed as a template. One syntactic approach to it is to create a template block that prints out {{:

description: "VALUE = {{ "{{" }} $value }}"
#                     ^^^^^^^^^^ a {{ ... }} block that prints "{{"