Search code examples
memory-managementapache-flinkbatch-processingtaskmanager

how to set flink taskmanager memory when submit a job


if I submit one job to flink, and I know the maximum memory required for this task.

so if I want to set the taskMangaer memory(maybe the default memory set by Flink is too big for me).

how can I adjust the taskmanager memory on my job? or only by modifying the source code can I achieve my purpose?


Solution

  • There is no API for configuring task managers from within a Flink job.

    Configuring task manager memory is done as part of launching the task manager. In the case of a session cluster, this is done independently from submitting a job. But in the case of a job or application cluster, it makes sense to configure the task manager(s) to fit the requirements of that specific job (or set of jobs).

    This is usually done in the cluster config (in conf/flink-conf.yaml), but you can also configure task manager memory via JVM parameters. The simplest approach is to set either taskmanager.memory.flink.size (memory for Flink itself) or taskmanager.memory.process.size (memory for the Flink JVM process).

    There's a section of the documentation devoted to memory configuration, and it goes into much more detail. https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/memory/mem_setup/ is the entry point to this section.

    While there is no API for configuring task managers, it is possible for a job to specify its requirements. See https://nightlies.apache.org/flink/flink-docs-stable/docs/deployment/finegrained_resource/.