Search code examples
parallel-processingapache-flinkflink-streaming

Flink CPU configuration


Official Flink Documentation states that for each core in your cpu, you have to allocate 1 slot and increase the parallelism level by one simultaneously.

One of my custom operators requires more than 1 CPU for computing (It is how it works in Heron). My system's cpu is 2.5. But Flink only uses 1 cpu. Do you know how can I config the Flink to use more CPU with only 1 slot?


Solution

  • "One slot per core" is merely a rule of thumb. Nothing enforces this.

    Each subtask (an instance of an operator chain) is single-threaded, but the slots within a task manager, and the task managers within a machine or container, will use all of the resources made available to them.