How to submit MR job to YARN cluster with ResourceManager HA wrt Hortowork's HDP?

Question

I am trying to understand how to submit a MR job to Hadoop cluster, YARN based.

Case1:

For case in which there is only one ResourceManager (that is NO HA), we can submit the job like this (which i actually used and I believe is correct).

hadoop jar word-count.jar  com.example.driver.MainDriver -fs  hdfs://master.hadoop.cluster:54310 -jt master.hadoop.cluster:8032 /first/dir/IP_from_hdfs.txt  /result/dir

As can be seen, RM is running on port 8032 and NN on 54310 and I am specifying the hostname becasue there is only ONE master.

case2:

Now, for the case when there is HA for both NN and RM, how do I submit the job? I am not able to understand this, because now we have two RM and NN (active / standby), and I understand that there is zookeeper to keep track of failures. So, from client perspective trying to submit a job, do I need to know the exact NN and RM for submitting the job or is there some logical naming which we have to use for submitting the job?

Can anyone please help me understand this?

Answer 1

With or without HA, the command to submit the job remains the same.

hadoop jar <jar> <mainClass> <inputpath> <outputpath> [args]

Using -fs and -jt is optional and are not used unless you want to specify a Namenode and JobTracker that is different from the one in the configurations.

If the fs.defaultFS property in core-site.xml and the properties defining the nameservice (dfs.nameservices) and its namenodes are configured properly in hdfs-site.xml of the client, the Active Master will be chosen whenever a client operation is performed.

By default, this Java class is used by the DFS Client to determine which NameNode is currently Active.

<property>
  <name>dfs.client.failover.proxy.provider.<nameserviceID></name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>