If I'm on the master node of a Hadoop cluster, I can go to yarn-site.xml and see the properties yarn.nodemanager.resource.memory-mb and yarn.nodemanager.resource.cpu-vcores, for instance, for the master node only.
If my worker nodes are a different type of machine than the master node (e.g., have more memory or cpu), they will obviously have different values for these same properties, but as far as I can tell, I can't see these values in any file on the master node.
Is there a way to obtain these values for the worker nodes without connecting to one of the machines and accessing its yarn-site.xml? The reason I pose this question is that I would like to be able to programmatically deploy jobs with configuration based on the profile of the cluster it's running on, but where these jobs are submitted from the master node (which is typically very small compared to the task nodes).
Edit: This page describes how to submit HTTP requests to the RM REST API for information about the cluster. In my yarn-site.xml is the property
<property>
<name>yarn.resourcemanager.address</name>
<value><http address>:8032</value>
</property>
but if I type curl http://<http address>:8032/ws/v1/cluster/metrics -XGET, for instance, it responds:
It looks like you are making an HTTP request to a Hadoop IPC port. This is not the correct port for the web interface on this daemon.
Is there another port I should be trying, or is there a different way I should be making the request?
The following works:
$ curl http://$(hostname):8088/ws/v1/cluster/metrics
{"clusterMetrics":{"appsSubmitted":0,"appsCompleted":0,"appsPending":0,"appsRunning":0,"appsFailed":0,"appsKilled":0,"reservedMB":0,"availableMB":116736,"allocatedMB":0,"reservedVirtualCores":0,"availableVirtualCores":16,"allocatedVirtualCores":0,"containersAllocated":0,"containersReserved":0,"containersPending":0,"totalMB":116736,"totalVirtualCores":16,"totalNodes":1,"lostNodes":0,"unhealthyNodes":0,"decommissionedNodes":0,"decommissioningNodes":0,"rebootedNodes":0,"activeNodes":1}}