Search code examples
apache-flinkflink-streaming

Share client object among custom Flink KeyedProcessFunction's


I have an object that is a client for an external service that I am grabbing data off of to join with another Kafka source which is an operator in the Flink topology.

What are some ways or tricks so that I don't have to create a new client within every single process function?


Solution

  • If you have one slot per TM, then I don't see how you can avoid incurring the cost of one client per sub-task.

    If you have multiple slots per TM, then you could (carefully) share the client (via a static class member) across all sub-tasks running in slots on the same TM.