Search code examples
apache-flinkdistributed-systemmaster-slavestream-processing

How to force Apache Flink using a modified operator placement?


Apache Flink distributes its operators on available, free slots on the JobManagers (Slaves). As stated in the documentation, there is the possibility to set the SlotSharingGroup for every operator contained in an execution. This means, that two operators can share the same slot, where they are executed later.

Unfortunately, this option only allows to share the same group but not to assign a streaming operation to a specific slot.

So my question is: What would be the best (or at least one) way to manually assign streaming operators to specific slots/workers in Apache Flink?


Solution

  • There is ongoing development work in this direction. In particular, see FLIP-56: Dynamic Slot Allocation. I don't know if this goes far enough to satisfy your goals, but at the very least the refactorings and extensions it brings should be helpful.

    For more details, see FLINK-14187 and related issues.