After upgrading the Flink v.1.16.1 (from v.1.13.2) I have seen the following logs:
2023-03-06 INFO org.apache.flink.runtime.metrics.dump.MetricQueryService [] - Some metrics will not be reported as the metric dump would exceed the maximum size of 10485504 bytes.
Is it possible to change maximum size? Or should i remove the some metrics from the flink application (to keep application healthly)?
Ok this is an old question but I want to answer since I also spent a lot of time on this issue.
I don't know the technical details but the problem is default akka.framesize is 10MB and if you have a lot of metrics, parallelism is high or you have a lot of tasks in a job , the network package which carries the all metric data can exceed this value.
You should either decrease the content of the network package or increase the network framesize.
So some solutions depending on your situation:
1- Decrease the parallelism.
2- Reduce the tasks in a job. For example I had a job consisted of 30 tasks. I split this into 3 jobs with 10 tasks each.
3- Increase akka.framesize : 20971520b in flink config.