"or" in Prometheus expression not working

Question

I'm trying to filter out two pods from an expression but it doesn't seem to be working see below:

"expr": "sum(kube_pod_status_phase{...namespace=~\"${names_prefix}|kube-system|monitoring\", pod!~\"pod-to-exclude1|pod-to-exclude1\"}) by (pod)",

I want this alert to trigger alerts for every other pod but not ""pod-to-exclude1 & pod-to-exclude1"

What am I doing wrong?

Answer 1

Pods are ephemeral in Kubernetes, as their name is dynamic, I'd say its better to use regex with pod name prefix to exclude in the alert config.

The below query will return all the pods except the specified two pods.

sum by (pod)(kube_pod_status_phase{pod!~"pod1-prefix.*|pod2-prefix.*"})