Search code examples
javamemoryapache-flink

How to increase Flink Memory size

I'm Trying to run a Job on Flink task manager and I'm getting this exception :

Initializing the input processing failed: Too little memory provided to sorter to perform task. Required are at least 12 pages. Current page size is 32768 bytes.

I've set heap size in both task and job manager's via flink-conf.yml , anything else I should change to increase the memory ?

taskmanager.heap.size: 4096m
taskmanager.memory.size: 4096m
jobmanager.heap.size: 2048m
Solution

  • The error message indicates that the sorter does not get enough memory pages. The reason is that the available managed memory is not sufficient. There are multiple ways to solve this problem:

    1. Increase the available memory for a TaskManager via taskmanager.heap.size
    2. Increase the fraction of the managed memory which is taken from taskmanager.heap.size via taskmanager.memory.fraction (per default it is 0.7)
    3. Decrease the page size via taskmanager.memory.segment-size
    4. Decrease the number of slots on a TaskManager since a reduced parallelism per TM will decrease the number of memory consumers on the TM (operators get a bigger share of the available memory)

    If you are running exclusively batch loads, then you should also activate taskmanager.memory.preallocate: true which will enable the memory allocation at start-up time. This is usually faster because it reduces the garbage collection pressure.

    Another comment concerning taskmanager.memory.size: This value always needs to be smaller or equal than taskmanager.heap.size since it specifies how much memory from the overall heap space will be used for managed memory. If this parameter is not specified, then Flink will take a fraction of the available heap memory for the managed memory (specified via taskmanager.memory.fraction).