Search code examples
apache-sparkpysparkprometheusmonitoringgraphite

Filter metrics sent to Graphite/Prometheus from Spark

I'm trying to monitor Spark. Spark pushes metrics to Graphite and Prometheus but it seems I can't choose the metrics it sends and there are so many.

I don't want to overload the Graphite and Prometheus instances.

Is there a way of filtering those metrics? In spark configuration or Graphite/Prometheus configuration?

Solution

  • This configuration will send only metrics that fit in the provided regex.

    "spark.metrics.conf.*.sink.graphite.regex"="optional_regex_to_send_matching_metrics"

    You can see it I the documentation.