Search code examples
alertprometheusprometheus-alertmanager

How can i write the prometheus complex alerting rule in simplified manner?


The expression of my alert rule is shown below , i need to write as many as 20 alert rules with same expression but the method_name and service_name varies from rule to rule making my alert.yaml file ugly can anyone specify simplified manner for writing these type of alerts

Expression:

( sum(rate(grpc_server_handling_seconds_bucket{endpoint="http",grpc_method="MethodName",grpc_service="ServiceName",grpc_type="unary",job="JobName",le="1",service="ServiceName"}[15m])) by (job)+sum(rate(grpc_server_handling_seconds_bucket{endpoint="http",grpc_method="MethodName",grpc_service="ServiceName",grpc_type="unary",job="JobName",le="5",service="ServiceName"}[15m])) by (job) ) / 2 /  sum(rate(grpc_server_handling_seconds_count{endpoint="http",grpc_method="MethodName",grpc_service="ServiceName",grpc_type="unary",job="JobNAme",service="ServiceName"}[15m])) by (job) < 0.9


Solution

  • If you want to alert on all method_names & service_names you can just skip adding these in label selector entirely

    (
      (
        sum by (job) (rate(grpc_server_handling_seconds_bucket{endpoint="http",grpc_type="unary",job="JobName",le="1",service="ServiceName"}[15m])) 
      +
        sum by (job) (rate(grpc_server_handling_seconds_bucket{endpoint="http",grpc_type="unary",job="JobName",le="5",service="ServiceName"}[15m]))
      ) / 2 
    /  
     sum by (job)(rate(grpc_server_handling_seconds_count{endpoint="http",grpc_type="unary",job="JobNAme",service="ServiceName"}[15m]))
    ) < 0.9
    

    Otherwise you can use some templating tool like ytt to generate the required rules for all different combinations of method_names & service_names