How does spark report / collect metrics

This defines several sinks, metrics and so on. But they are collected?

Let say that I added JxmSink into metric.properties file and enable all instance metrics (master, applications, worker, executor, driver, shuffleService, applicationMaster).
Let say that jmx port is set.

Where to collect metrics: should I connect to all cluster nodes or only to driver node?

Solution

Spark metrics are not required to pull from individual nodes, if respective sink host configured in metric properties file, then metrics will be pushed to it for every configured seconds. Our setup configured to have GraphiteSink to collect the metrics, required configuration for the same as detailed below (along with others you mentioned)

Prepare a metrics configuration properties file with Graphite server end point

    *.sink.graphite.class=org.apache.spark.metrics.sink.GraphiteSink
    *.sink.graphite.host=<graphite-server-host>
    *.sink.graphite.port=<graphite-server-port>
    *.sink.graphite.period=10
    *.sink.graphite.prefix=dev

Make sure metrics properties file being passed into --files option in spark-submit job script, so that it will be used by executor nodes for sending metrics