Search code examples
prometheusprometheus-alertmanagerprometheus-operator

how to exclude some job in count by prometheus query


I have a prometheus query which is as below:

100 * (count by(job, namespace, service) (up == 0) / count by(job, namespace, service) (up)) > 10

and it is giving result

{job="kubernetes-pods"}
12.121212121212121
{job="kube-prometheus-kube-proxy", namespace="kube-system", service="kube-prometheus-kube-proxy"}  12.98
{job="kubernetes-service-endpoints"}   19

but in the result i would like to exclude

job="kube-prometheus-kube-proxy"

How do i do that ? I tried something like :

100 * (count by(job {job!=“kube-prometheus-kube-proxy”}, namespace, service) (up == 0) / count by(job {job!=“kube-prometheus-kube-proxy”},, namespace, service) (up)) > 10

and 

100 * (count by(job, namespace, service) {job!=“kube-prometheus-kube-proxy”} (up == 0) / count by(job, namespace, service) {job!=“kube-prometheus-kube-proxy”} (up)) > 10

and other options too.. But none of them work as expected... In the original count by query, what needs to be changed so that it won't includ job job="kube-prometheus-kube-proxy" in result ?


Solution

  • The filter has to be written after metric name, which is up in this case. Thus:

    100 * (
      count by(job, namespace, service) (up{job!="kube-prometheus-kube-proxy"} == 0)
      / 
      count by(job, namespace, service) (up{job!="kube-prometheus-kube-proxy"})
    ) > 10
    

    Also, it seems my Prometheus does not like your stylish quotes. I replaced them with typical ".