Combining alert rules for all metrics

Question

I have a alert.rules.yml file which looks like this

groups:
  - name: my-alert-rules
    rules:
      - alert: FileCountRANRRCTooHigh
        expr: ((file_count_RAN_RRC - file_count_RAN_RRC offset 24h) / (file_count_RAN_RRC offset 24h) * 100) > 80
        for: 30s
        labels:
          severity: warning
        annotations:
          summary: "File count for {{ $labels.folder }} increased significantly"
          description: "The file count for {{ $labels.folder }} increased by over 80% in the last 24 hours."

      - alert: FileCountGiTooHigh
        expr: ((file_count_Gi - file_count_Gi offset 24h) / (file_count_Gi offset 24h) * 100) > 80
        for: 30s
        labels:
          severity: warning
        annotations:
          summary: "File count for {{ $labels.folder }} increased significantly"
          description: "The file count for {{ $labels.folder }} increased by over 80% in the last 24 hours."

      - alert: FileCountGnTooHigh
        expr: ((file_count_Gn - file_count_Gn offset 24h) / (file_count_Gn offset 24h) * 100) > 80
        for: 30s
        labels:
          severity: warning
        annotations:
          summary: "File count for {{ $labels.folder }} increased significantly"
          description: "The file count for {{ $labels.folder }} increased by over 80% in the last 24 hours."

      - alert: FileCountRANVSETooHigh
        expr: ((file_count_RAN_VSE - file_count_RAN_VSE offset 24h) / (file_count_RAN_VSE offset 24h) * 100) > 80
        for: 30s
        labels:
          severity: warning
        annotations:
          summary: "File count for {{ $labels.folder }} increased significantly"
          description: "The file count for {{ $labels.folder }} increased by over 80% in the last 24 hours."

I am wondering if there's a better way to write this since the expression for all rules is similar

((file_count_{{ $labels.folder }} - file_count_{{ $labels.folder }} offset 24h) / (file_count_{{ $labels.folder }} offset 24h) * 100) > 80 

and have only one block under rules

Answer 1

In Prometheus name of metric in selector can be address in two ways: either in form name{} or through built-in label __name__.

So selector {name =~ "file_count_.+"} will get you all metrics with name starting with file_count_.

Since you report that your metrics have additional labels folder you can simply use expression ({name =~ "file_count_.+"} - {name =~ "file_count_.+"} offset 24h) / ({name =~ "file_count_.+"} offset 24h) * 100 > 80 in a single rule, and it will address all the metrics with the name starting with file_count_ at the same time.

---

Additionally, I'd like to repeat here, that root of your problem is that you didn't follow recommendations on metric naming. They should never be procedurally generated. If you would expose them properly with the same name and folder as a label, query for this kind of rule will be straightforward, and wouldn't require duplication of the exposition side.