Azure Application Insights AKS - How to Create 1 custom alert rule which will trigger multiple alerts, but with different names (per pod name)?

Question

Is there a way to create 1 custom alert rule for CrashLoopBackOff (as an example) which will be triggering multiple alerts. All alerts will be identical, but pod names will be different.

In Prometheus we receive alerts by each pod, but I`m not sure if it is possible to achieve in Azure Application Insights.

Answer 1

You can use “Alert rule” to archive it. Insight itself does not have feature to alert.
Steps:

1. Go to “Alerts” in your AKS, then “Create” -> “Alert rule”
2. Set signal name as “Custom log search”
3. Modify the query based on the following:

KubePodInventory
| where ContainerStatus  == 'waiting' 
// | where Name contains "microsoft-defender-"
| extend ContainerLastStatus=todynamic(ContainerLastStatus)
| summarize RestartCount = arg_max(ContainerRestartCount, Computer, Namespace, ContainerStatusReason) by Name

(Note: uncomment the 3rd line: “| where Name contains "microsoft-defender-"”, modify the name to whatever you want.)

4. Run one time, then “Continue Editing Alert”
5. Select the “Aggregation granularity”
6. Set alert logic as “Greater than 0”
7. Set “Split by dimensions”, set “ContainerStatusReason” with any value you want
   
8. Do whatever you want next. Full document: https://learn.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-create-log-alert-rule
   Example output:
   
9. Did the same steps for other pods/deployments you want. Set different alert name to differentiate different alerts for different pods/deployments.
   (Note: in my example, "testcrashcontainer" is the alert name for the target pods I want to monitor. Set different name as you want.)