Search code examples
apache-flinkhadoop-yarn

Submiting Flink Job on YARN Issue

I want to submit my flink job on YARN with this command:

./bin/flink run -m yarn-cluster -p 4 -yjm 1024m -ytm 4096m ./task.jar

but I faced this error:

is running beyond virtual memory limits. Current usage: 390.3 MB of 1 GB physical memory used; 2.3 GB of 2.1 GB virtual memory used. Killing container.
Solution

  • This is caused because of the a variable named yarn.nodemanager.vmem-pmem-ratio which is default set to 2.1, in this command this ratio is 4096/1024 = 4

    You have 3 ways:

    1 - If you have access to YARN configuration you can set yarn.nodemanager.vmem-check-enabled is yarn-site.xml to false.

    2 - Another if you have access to configuration is to change the ratio value from 2.1 to 5 e.g.

    3 - If you don't have access you can change the YARN configurations, you can change the ytm and yjm values in order to satisfy the ratio condition, for example: -yjm 4096 -ytm 4096.