We have been running a production grade system where we want to start a secondary namenode in AWS EMR automatically. Below is the output of jps in which secondary namenode daemon is not running
[root@ip-10-2-23-23 ~]# jps
6241 Bootstrap
7041 ResourceManager
10754 RunJar
6818 WebAppProxyServer
10787 SparkSubmit
7619 JobHistoryServer
6922 ApplicationHistoryServer
3661 Main
4877 Main
6318 NameNode
8943 LivyServer
4499 Jps
5908 Bootstrap
4791 Main
10619 StatePusher
9918 HistoryServer
Secondary namenode is required to do namenode checkpointing and do regular creation of fsImage .I have not configured any HA for Namenode.
Command we ran manually to create FsImage is
hdfs secondarynamenode -checkpoint
How a secondary namenode can be started in AWS EMR or there is any configuration ?
Hadoop version : Hadoop 2.8.3-amzn-0
AWS EMR doesn't run secondary Namenode proces on EMR so FSImage won't be created, Running a cron every hour to create a FSImage solves the problem of too much disk usage because the FSImage merges the snapshot (Namenode metadata) to create a new FsImage of smaller size. FSImage creation is a costly operation for Namenode and it utilizes instance resources. If there are too many snapshots pending for merging , Namenode may never recover from this tedious process , so it is better to create FSImage frequently via cron .In a standard Hadoop system this job is done by running a secondary Namenode on a separate instance but EMR doesn't have concept of two masters so Master node is always a single point of failure.
hdfs secondarynamenode -checkpoint
Other solution to this problem is running EMR on custom Hadoop like MapR .