Search code examples
grafanagrafana-alertsgrafana-templating

How to pass pod_name in pod_down grafana alert message


I created the pod_down alert rule and this is my query:

sum by(namespace,pod) (changes(kube_pod_status_ready{condition="true"}[5m]))

1. alert rule query

2. alert rule classic conditiion

3. alert rule message template

I want to pass pod_name into the alert message template.

pod= nginx-3yh536hs-kd pod was down

environment = dev

namespace = default

summary = pod_down alert rule.


@markalex So, I found the simple template and tried it for the discord alert message (Contact points) but still, I can't get the correct output

template

{{ define "default_template" }}
  **[{{ index .Annotations "summary" }}]**
  **Environment:** {{ index .Labels "environment" }}
  **Pod:** {{ index .Labels "pod" }}
  **Type:** {{ index .Labels "type" }}
  
  **Error Message:**
  {{ .Annotations.summary }}
{{ end }}

discord alert message

**[
[RESOLVED] Pod Down Alert Development (postgresql-backup dev postgresql-backup k8s)

Grafana v10.1.5
Spidey Bot
BOT
 — Today at 10:07 AM
**[
[FIRING:1]  (TestAlert Grafana)

Grafana v10.1.5

Solution

  • Finally, I got my correct alert message template.

    Pod CPU high alert

    sum(rate(container_cpu_user_seconds_total{container!="pod",namespace="default",container!=""}[5m])) by (pod) * 1024
    

    alert condition image

    Add Summary into anntations

    The host **{{ index $labels "pod" }}** has exceeded 500Mi CPU usage for the last 5 minutes
    

    and save rule and exit

    create default message template.

    {{ define "slack.title_NEW_1702356284919" }}
    {{ len .Alerts.Firing }} firing alert(s), {{ len .Alerts.Resolved }} resolved alert(s)
    {{ end }}
    
    {{ define "slack.print_alert" -}}
    [{{.Status}}] {{ .Labels.alertname }}
    Labels:
    {{ range .Labels.SortedPairs -}}
    - {{ .Name }}: {{ .Value }}
    {{ end -}}
    {{ if .Annotations -}}
    Annotations:
    {{ range .Annotations.SortedPairs -}}
    - {{ .Name }}: {{ .Value }}
    {{ end -}}
    {{ end -}}
    {{- end }}
    
    {{ define "slack.message" -}}
    {{ if .Alerts.Firing -}}
    {{ len .Alerts.Firing }} firing alert(s):
    {{ range .Alerts.Firing }}
    {{ template "slack.print_alert" . }}
    {{ end -}}
    {{ end }}
    {{ if .Alerts.Resolved -}}
    {{ len .Alerts.Resolved }} resolved alert(s):
    {{ range .Alerts.Resolved }}
    {{ template "slack.print_alert" .}}
    {{ end -}}
    {{ end }}
    {{- end }}
    

    output discord alert message