Prometheus K8s extraConfigmapMounts fails to load multiple files to the same directory

Question

Prometheus Question: I am using prometheus on Helm, and I want to mount several .yml files on the same location /etc/config/alrtingRules It is vital that these remain separated as different files in the git repo. I have tried mounting them each to its own configMap, and use "extraConfigmapMounts" to put them all in location, but I am facing difficulties.

I've tried two configurations:

first:

  extraConfigmapMounts:
    - name: recording-rules
      mountPath: /etc/config/recording-rules.yml
      subPath: recording-rules.yml
      configMap: recording-rules
      readOnly: true
    - name: dummytest-alerting
      mountPath: /etc/config/alertingRules/dummytest.yml
      subPath: dummytest.yml
      configMap: dummytest-alerting
      readOnly: true
    - name: app1-alerting
      mountPath: /etc/config/alertingRules/app1.yml
      subPath: app1.yml
      configMap: app1-alerting
      readOnly: true
    - name: app2-alerting
      mountPath: /etc/config/alertingRules/app2.yml
      subPath: app2.yml
      configMap: app2-alerting
      readOnly: true

This helm succeeds, but then prometheus-server fails to load with this error:

ts=2022-06-13T08:25:35.322Z caller=manager.go:968 level=error component="rule manager" msg="loading groups failed" err="/etc/config/alertingRules/dummytest.yml: read /etc/config/alertingRules/dummytest.yml: is a directory

second:

  extraConfigmapMounts:
    - name: recording-rules
      mountPath: /etc/config/
      subPath: recording-rules.yml
      configMap: recording-rules
      readOnly: true
    - name: dummytest-alerting
      mountPath: /etc/config/alertingRules/
      subPath: dummytest.yml
      configMap: dummytest-alerting
      readOnly: true
    - name: app1-alerting
      mountPath: /etc/config/alertingRules/
      subPath: app1.yml
      configMap: app1-alerting
      readOnly: true
    - name: app2-alerting
      mountPath: /etc/config/alertingRules/
      subPath: app2.yml
      configMap: app2-alerting
      readOnly: true

With this configuration, Helm fails giving this error:

Error: UPGRADE FAILED: failed to create patch: The order in patch list: [map[mountPath:/etc/config name:server-recording-rules readOnly:true subPath:recording-rules.yml] map[mountPath:/etc/config name:config-volume] map[mountPath:/data subPath:] map[mountPath:/etc/config/alertingRules name:server-app2-alerting readOnly:true subPath:app2.yml] map[mountPath:/etc/config/alertingRules name:server-app1-alerting readOnly:true subPath:app1.yml] map[mountPath:/etc/config/alertingRules name:server-dummytest-alerting readOnly:true subPath:dummytest.yml]] doesn't match $setElementOrder list: [map[mountPath:/etc/config] map[mountPath:/data] map[mountPath:/etc/config] map[mountPath:/etc/config/alertingRules] map[mountPath:/etc/config/alertingRules] map[mountPath:/etc/config/alertingRules]]

Any suggestions as to how to mount several .yml files into the /etc/config of prometheus server?

Answer 1

The solution I found is thus: I create an empty configMap called "alerting-rules". Every microservice gets its own yaml file with only "data" field. For example:

data:
  app1.yaml: |
    groups:
    ...
    ...
    ...
    ...
    ...

I mount the various alerting yamls into alerting-rules using the "kubectl patch" command, for example:

- kubectl patch configmap -n prometheus alerting-rules --patch-file path/to/app1.yaml

In the prometheus values I add this:

server:
  extraConfigmapMounts:
    - name: recording-rules
      mountPath: /etc/config/recordingRules
      configMap: recording-rules
      readOnly: true
    - name: alerting-rules
      mountPath: /etc/config/alertingRules
      configMap: alerting-rules
      readOnly: true

And further down, under I add this:

serverFiles:
  prometheus.yml:
    rule_files:
      - /etc/config/recordingRules/recording-rules.yaml
      - /etc/config/alertingRules/dummytest.yaml
      - /etc/config/alertingRules/app1.yaml
      - /etc/config/alertingRules/app2.yaml

This loads all alerts into prometheus, thus allowing us to manage our alerts in a per-microservice yaml.