Apply alert config for prometheus Alertmanager which is runnning on the kubernetes

Question

I have a kubernetes cluster which running on EKS

I already installed prometheus and alertmanager from this repository.

This is the set of tools, so I think it is useful for a begginner like me.

Now I can see prometheus dashboad via localhost:9090 and my pods are here.

There are somehow three alertmanager.(Maybe same as node?)

$kubectl get pod -A
NAMESPACE     NAME                                   READY   STATUS    RESTARTS   AGE
kube-system   aws-node-669m7                         1/1     Running   0          29m
kube-system   aws-node-7rwhg                         1/1     Running   0          29m
kube-system   aws-node-f7tnf                         1/1     Running   0          39m
kube-system   cluster-autoscaler-78c99d7c59-6gbn7    1/1     Running   0          31m
kube-system   coredns-7dbf6bcd5b-5hgjx               1/1     Running   0          49m
kube-system   coredns-7dbf6bcd5b-ff5b4               1/1     Running   0          49m
kube-system   kube-proxy-jrwxh                       1/1     Running   0          29m
kube-system   kube-proxy-lwm9q                       1/1     Running   0          39m
kube-system   kube-proxy-xp9l9                       1/1     Running   0          29m
monitoring    alertmanager-main-0                    2/2     Running   0          35m
monitoring    alertmanager-main-1                    2/2     Running   0          35m
monitoring    alertmanager-main-2                    2/2     Running   0          35m
monitoring    blackbox-exporter-69f4d86566-d99bk     3/3     Running   0          36m
monitoring    grafana-79cd8d4b69-2k9np               1/1     Running   0          36m
monitoring    kube-state-metrics-56f8746666-q298b    3/3     Running   0          31m
monitoring    node-exporter-ndln7                    2/2     Running   0          29m
monitoring    node-exporter-psqng                    2/2     Running   0          29m
monitoring    node-exporter-z92vx                    2/2     Running   0          35m
monitoring    prometheus-adapter-77f56b865b-4sgv7    1/1     Running   0          31m
monitoring    prometheus-adapter-77f56b865b-f9tx6    1/1     Running   0          35m
monitoring    prometheus-k8s-0                       2/2     Running   0          35m
monitoring    prometheus-k8s-1                       2/2     Running   0          35m
monitoring    prometheus-operator-54f6687b97-5mdbb   2/2     Running   0          35m

then, What I want to do is set the alert which send the email to my gmail account when nodes are more than 4.

so I made the alertmanager.yml (It is not the alert what I meant, this is just the sample, just practicing.)

global:
  resolve_timeout: 5m
  smtp_from: 'prometheus.test@example.com'
  smtp_smarthost: 'localhost:25'
route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'web.hook'
  routes:
  - receiver: 'mailtest'
    match:
      test_type: SampleAlert
receivers:
- name: 'mailtest'
  email_configs:
    - to: 'root@localhost.localdomain'
      require_tls: false
- name: 'web.hook'
  webhook_configs:
  - url: 'https://127.0.0.1:5001/'
inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

Then document says, I can attache this as nohup ./alertmanager --config.file=alertmanager.yml &

however, I don't have alertmanager command in my local PC.

How should I apply this yml to the alertmanager pods running on EKS?

Answer 1

As mentioned in docs: You can add Alert Manager configuration in your values.yaml file under values.alertmanager.config Alert manager is imported to kubernetes as a secret, if you want to se what is being applied, you need to run kubectl get secrets then extract data from it.

To do more complex manipulations on your configurations, you need to get familiar with libsonnet .