I am aware to set the memory and vcores in YARN using the following properties:
- yarn.nodemanager.resource.memory-mb
- yarn.nodemanager.resource.cpu-vcores
I have a heterogenous YARN cluster with nodes having following configuration:
- Node1 (8cores, 16GB RAM)
- Node2 (8cores, 16GB RAM)
- Node3 (32cores, 64GB RAM)
- Node4 (32cores, 64GB RAM)
I want to set the nodemanager memory and cores to be different for Node1,Node2 and Node3,Node4.
Node1 & Node2
- yarn.nodemanager.resource.memory-mb = 10240
- yarn.nodemanager.resource.cpu-vcores = 15
Node3 & Node4
- yarn.nodemanager.resource.memory-mb = 40240
- yarn.nodemanager.resource.cpu-vcores = 25
How to achieve this with/without using Ambari?
Without Ambari:
You can achieve the heterogeneous resource assignment directly by configuring the above properties in yarn-site.xml
of individual nodes and restarting the YARN services.
With Ambari:
In Ambari, You can create Configuration Groups
for individual nodes
of cluster from the Manage Configuration Group
link visible on the Configs
tab next to Group drop down
. While creating the configuration groups you will find option of selecting the node where you wants to apply the configuration. Once configurations gets created for individual nodes, changes can be applied by restarting the YARN services. Follow Heterogeneous Configuration to know how it can be configured.