Customize Flink - Prometheus metrics

Question

I need to export custom metrics from Flink 1.10 to Prometheus. I have my custom metrics already created and working, but the issue is that when I print out (in terminal for example) to see the metrics, a lot of metrics comes out from Flink and I don't need them, such as: flink_taskmanager_job_task_Shuffle_Netty_Input_Buffers_inputQueueLength, and many more. I'm just interest into spread from Flink my custom metrics to Prometheus, and remove the rest of them. So, questions:

1. Is there anyway to remove all the metrics exported from Flink and just keep my custom metrics to Prometheus?
2. Is there anyway to create statics task_id to not accumulate a lot of information in Prometheus? Because I supposed that that ids are not fixed and with every changes in the application that requires a stop/start, Flink will create a new task_id.

I've been able to remove a few tags using: "metrics.reporter.cep_reporter.scope.variables.excludes":"job_id;job_name;task_attempt_id;task_attempt_num;task_name;operator_id;operator_name;subtask_index;tm_id;host;Netty"

but is not enough, there are more than 800 metrics that I don't need, JVM for example, I'm using another node_exporter to scrape those metrics, need to remove this metrics too.

Any help will be appreciated. Thanks a lot.

Answer 1

Disclaimer: I haven't tried this.

What I would try would be to set a user scope on your custom flink metrics, and then configure prometheus to only scrape those metrics.