Search code examples
kubernetesprometheusprometheus-alertmanager

How to run pod based on Prometheus alert


Is there any way we can run pod based on the alert fired from Prometheus? We have a scenario where we need to execute a pod based on the disk pressure threshold. I am able to create alert but I need to execute a pod. How can I achieve that?

groups:
  - name: node_memory_MemAvailable_percent
    rules:
    - alert: node_memory_MemAvailable_percent_alert
      annotations:
        description: Memory on node {{ $labels.instance }} currently at {{ $value }}% 
          is under pressure
        summary: Memory usage is under pressure, system may become unstable.
      expr: |
        100 - ((node_memory_MemAvailable_bytes{job="node-exporter"} * 100) / node_memory_MemTotal_bytes{job="node-exporter"}) > 80
      for: 2m
      labels:
        severity: warning

Solution

  • Yes we have webhook but service we implemented by using am executor as custom service from am executor custom script we have run the required job from ado pipeline