Search code examples
apache-flinkflink-streaming

How to get metrics of all parallelisms in Flink?

In flink web UI, I can get metrics of each parallelism,

for parallelism 0, it likes:

0_filter_numberOfRecords in

for parallelism 9, it likes:

9_filter_numberOfRecords in

How to get the same metric of all parallelism?

Solution

  • You can obtain the aggregated metrics by directly querying Flink's rest endpoint. You need the following information:

    1. Job id jobId identifying the job to which the operator belongs you want to query
    2. Vertex id vertexId identifying the operator from which you want to retrieve the metrics

    The jobId can be obtained by querying http://flink_cluster:port/jobs.

    The vertexId can be obtained by querying http://flink_cluster:port/jobs/:jobId which gives you the job information (list of operators with their vertex ids).

    Having obtained the required ids, you can send a GET request to:

    http://flink_cluster:port/jobs/:jobId/vertices/:vertexId/subtasks/metrics?get=filter.numberOfRecords

    Please replace :jobId and :vertexId with the respective ids.

    If you want to aggregate the metrics over a sub range of sub tasks, then you can change your request to subtasks/metrics?get=filter.numberOfRecords&subtasks=0-1,4-5 in order to specify the subtasks to include in the aggregation.