Search code examples
memory-managementhadoop-yarnambari

Configuring NodeManager memory and vcores in a heterogenous YARN cluster?


I am aware to set the memory and vcores in YARN using the following properties:

  • yarn.nodemanager.resource.memory-mb
  • yarn.nodemanager.resource.cpu-vcores

I have a heterogenous YARN cluster with nodes having following configuration:

  • Node1 (8cores, 16GB RAM)
  • Node2 (8cores, 16GB RAM)
  • Node3 (32cores, 64GB RAM)
  • Node4 (32cores, 64GB RAM)

I want to set the nodemanager memory and cores to be different for Node1,Node2 and Node3,Node4.

Node1 & Node2

  • yarn.nodemanager.resource.memory-mb = 10240
  • yarn.nodemanager.resource.cpu-vcores = 15

Node3 & Node4

  • yarn.nodemanager.resource.memory-mb = 40240
  • yarn.nodemanager.resource.cpu-vcores = 25

How to achieve this with/without using Ambari?


Solution

  • Without Ambari:

    You can achieve the heterogeneous resource assignment directly by configuring the above properties in yarn-site.xml of individual nodes and restarting the YARN services.

    With Ambari:

    In Ambari, You can create Configuration Groups for individual nodes of cluster from the Manage Configuration Group link visible on the Configs tab next to Group drop down. While creating the configuration groups you will find option of selecting the node where you wants to apply the configuration. Once configurations gets created for individual nodes, changes can be applied by restarting the YARN services. Follow Heterogeneous Configuration to know how it can be configured.