Search code examples
hadoophdfscluster-computinghostssystem-administration

How to get number of hosts in Hadoop Cluster, their IP and rack

I'm working on a cluster but I don't know how many hosts it has exactly, which are their IPs and what rack they belong to. I've previously worked with clusters managed via Cloudera and got that information from the cloudera api (http://cloudera.github.io/cm_api/apidocs/v16/), in particular this (http://cm_server_host:7180/api/v16/hosts) gave me all the info I was looking for. But how can I do that if the cluster doesn't use Cloudera? It has spark as well, but since there is Hadoop and HDFS I think the information is more likely to be found there.

Thanks in advance!

Solution

  • You can find those information via http api, that by default should be available under this url:

    http://<namenodehost>:50070

    and via YARN http api, that by default should be available under this url: 

    http://<resourcemanagerhost>:8088/cluster/nodes

    Alternatively you can use ResourceManager REST API’s. 

    http://<resourcemanagerhost>:8088/ws/v1/cluster/nodes

    More about the topic you can find for example here:

    https://www.datadoghq.com/blog/collecting-hadoop-metrics/