I am using Hadoop cluster with 9 nodes. I would like to know what is the basic datanode configuration in Hadoop cluster.
I am using following configuration on Namenode and Datanode.
RAM = 4GB
Cores = 4
Disk = 8 ( Total 16GB storage space)
Running sample sort and word count jobs to check the hadoop network perfromance.
Is the configuration I have chosen is right ?
Thanks & Regards,
Abdul Navaz
In theory you can use any configuration you want, as long as it is configured properly (so that Hadoop knows the server capabilities etc).
In practice you should have at least 4GB of RAM per server (according to my experience). Number of cores influences the performance and you should have enough free disk space for running the server (set the amount of space reserved for non-HDFS things (such as server work) with settings dfs.datanode.du.reserved in hdfs-site.xml).
I am unsure how much free space you should have for normal Hadoop work - I guess it depends on your jobs. 16GB is not very much in any case.
Generally speaking, your setup should be just fine as long as it is configured properly.