Provisioned Grafana Alerts send alerts when "no_data_state": "OK" is configured

Question

Grafana version 9.2.3

I've created alerts via the Grafana UI and set the error and noData states as OK, meaning don't send any alerts if the alert is in an "error" or "no data" state.

These alerts work perfectly and remain in a state of "OK" or "OK (no Data)". But then when provisioning (in a configmap.yaml file) what seems like an identical alert to the one configured in the UI, the alert is triggered for a "no Data" state.

I've set the following parameters

"no_data_state": "OK",
"exec_err_state": "OK",

in the full config file below.

apiVersion: v1
kind: ConfigMap
metadata:
  name: alerting-config
data:
  config.json: |
    {
      "contactPoints": [
        {
          "orgId": 1,
          "name": "Pagerduty",
          "receivers": [
            {
              "uid": "first_uid",
              "type": "pagerduty",
              "settings": {
                "integrationKey": "xxxxxxxxx",
                "severity": "warning",
                "component": "Grafana"
              }
            }
          ]
        }
      ],
      "groups": [
        {
          "orgId": 1,
          "name": "API Metrics",
          "folder": "API Metrics",
          "interval": "60s",
          "rules": [
            {
              "uid": "my_id_1",
              "title": "TEST Alert 500 Response",
              "condition": "B",
              "no_data_state": "OK",
              "exec_err_state": "OK",
              "data": [
                {
                  "refId": "C",
                  "queryType": "",
                  "relativeTimeRange": {
                    "from": 600,
                    "to": 0
                  },
                  "datasourceUid": "xxxxxxxx",
                  "model": {
                    "editorMode": "builder",
                    "expr": "rate(django_http_responses_total_by_status_created{status=\"500\"}[10m])",
                    "hide": false,
                    "intervalMs": 1000,
                    "legendFormat": "__auto",
                    "maxDataPoints": 43200,
                    "range": true,
                    "refId": "C"
                  }
                },
                {
                  "refId": "D",
                  "queryType": "",
                  "relativeTimeRange": {
                    "from": 0,
                    "to": 0
                  },
                  "datasourceUid": "-100",
                  "model": {
                    "conditions": [
                      {
                        "evaluator": {
                          "params": [
                            0,
                            0
                          ],
                          "type": "gt"
                        },
                        "operator": {
                          "type": "and"
                        },
                        "query": {
                          "params": [
                            "C"
                          ]
                        },
                        "reducer": {
                          "params": [

                          ],
                          "type": "sum"
                        },
                        "type": "query"
                      }
                    ],
                    "datasource": {
                      "name": "Expression",
                      "type": "__expr__",
                      "uid": "__expr__"
                    },
                    "hide": false,
                    "intervalMs": 1000,
                    "maxDataPoints": 43200,
                    "refId": "D",
                    "type": "classic_conditions"
                  }
                }
              ],
              "for": "5m",
              "annotations": {
                "description": "500 response has been triggered in the API. Please investigate.",
                "summary": "A 500 response has been triggered in the API"
              },
              "labels": {
              }
            }
          ]
        }
      ]
    }
            

I'm not sure why the alert that has been provisioned in the config file is triggering an alert but the alert configured in the UI works perfectly?

Any help would be massively appreciated :) Cheers!

Answer 1

You are missing the field below in your configmap.yaml file:

# <string> the state the alert rule will have when no data is returned
#          possible values: "NoData", "Alerting", "OK", default = NoData
noDataState: OK

As outlined here: https://grafana.com/docs/grafana/latest/administration/provisioning/