Search code examples
prometheusprometheus-alertmanager

Inhibition with exceptions (negative matchers)

I have a problem with inhibition rules because of exceptions to inhibition rules.

For example, we have 50 teams, and every team handles their alerts. When a data center goes down (e.g. because of network problems), we want to inhibit all alerts except for Team_1234567890 and Team_ABCDEFGHIJ.

Problem is that Alertmanager doesn't support negative matchers for inhibition: Negative matchers for routing and inhibition #1023 - https://github.com/prometheus/alertmanager/issues/1023

Golang, and Prometheus/Alertmaneger, doesn't support "?!" negative look ahead in regex: https://github.com/google/re2/wiki/Syntax

How to setup inhibition rules for this example?

Thanks, Denis

Solution

  • Julien Pivotto (roidelapluie/Github) has written solution to this use case: https://github.com/prometheus/alertmanager/issues/1023#issuecomment-671851280

    You could use prometheus

    alerting:
  alert_relabel_configs:
  - source_labels: [team]
    regex: Team_1234567890|Team_ABCDEFGHIJ
    target_label: dc_team_alert
    replacement: "yes"

    and inhibit

    target_match:
   dc_team_alert: ""