My k8's cluster has dev, prod, qa environment and have created namespace according to available environments. Each pod will be having a label of the environment. I have created grafana dashboards and alerts for node, service, pod, PVC. The alerts are configured in alertmanager(Prometheus). The issue I'm facing right now is that the alerts must be triggered according to the environment. If an alert is triggering, I don't know from which environment it gets triggered. So what can be done?
If the alerts must be triggered according to the environment, you must include the environment label in your service, pod or PVC metrics.
The following solution for adding pod labels to kubelet metrics: https://stackoverflow.com/a/70196329/3781502
may serve this purpose.
Not possible for node alerts, as node is a kubernetes cluster level resource.