Search code examples
apache-sparkdockerfileprometheusjmx-exporterspark-operator

How to change prometheus jmx exporter rules in spark-operator image working as extension of gcr.io/spark-operator/spark:v3.0.0-hadoop3

We are trying to substitute the prometheus.yaml in /etc/metrics/conf as it hosts older rules. We try to copy new rules file:

RUN mkdir -p /etc/metrics/conf
COPY conf/prometheus.yaml /etc/metrics/conf/prometheus.yaml

But the rules are not picked up and prometheus still ignores the new metrics.

Solution

  • Sad truth is that in fact the folder and file is alredy there (looks like a symlink to /etc/metrics/conf/..data/prometheus.yaml location).

    We had to work it around with

    1. copying files to new location

      COPY conf/prometheus.yaml /etc/metrics/conf2/prometheus.yaml

    2. add configFile parameter to the helm chart:

      configFile: "/etc/metrics/conf2/prometheus.yaml"

    That causes javaagent's properties passed to jmx-exporter to use this new prometheus.yaml location