Search code examples
javaapache-flinkdashboardmetrics

Flink Dashboard: Operator Throughput

  1. I have a program that uses flink (1.9) and I want to check the throughput of instances of a Map operator with the help of the dashboard. From the already existing metrics numRecordsInPerSecond seems to be the most promising, but I guess it doesn't involve processing time. Am I wrong?

  2. I've defined my own metric (throughput) that essentially calculates the average throughput by dividing the number of records processed by the total execution time of the OUT map(IN value) function. But this does not count anything that happens outside the map function.

  3. Another idea would be to add a meter at the end of the map function, but I suppose if the source doesn't produce records fast enough the throughput calculated will be worse just because the operator remains idle a lot of the time. Is this correct?

Please specifically answer 1 and 2. Also, how do you usually calculate the throughput in your programs?

enter image description here

Solution

  • All of Flink's Meter metrics, such as numRecordsInPerSecond, are measuring rates in terms of processing time.

    I'm generally content to rely on these built-in metrics for measuring throughput. But you might want to add a custom metric in the sink, since Flink always returns 0 for numRecordsOut and numRecordsOutPerSecond for sinks.