Is there any way we can run pod based on the alert fired from Prometheus? We have a scenario where we need to execute a pod based on the disk pressure threshold. I am able to create alert but I need to execute a pod. How can I achieve that?
groups:
- name: node_memory_MemAvailable_percent
rules:
- alert: node_memory_MemAvailable_percent_alert
annotations:
description: Memory on node {{ $labels.instance }} currently at {{ $value }}%
is under pressure
summary: Memory usage is under pressure, system may become unstable.
expr: |
100 - ((node_memory_MemAvailable_bytes{job="node-exporter"} * 100) / node_memory_MemTotal_bytes{job="node-exporter"}) > 80
for: 2m
labels:
severity: warning
Yes we have webhook but service we implemented by using am executor as custom service from am executor custom script we have run the required job from ado pipeline