Search code examples
hadoophadoop-yarnemramazon-emr

Hadoop (EMR) Cluster Fair Scheduler is completing FIFO instead of in Parallel

This is my first time attempting to configure the YARN scheduler and it is not working as I would hope. The cluster originally worked as FIFO and I am attempting to get jobs to run in parallel. I have added to the top of the yarn-site.xml

  <property>
    <name>yarn.resourcemanager.scheduler.class</name>
    <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler</value>
  </property>


  <property>
    <name>yarn.scheduler.fair.allocation.file</name>
    <value>/etc/hadoop/conf.empty/fair-scheduler.xml</value>  
  </property>

And then added the file /etc/hadoop/conf.empty/fair-scheduler.xml:

<allocations>
    <queue name="root">
        <weight>1.0</weight>
        <schedulingPolicy>fair</schedulingPolicy>
        <aclSubmitApps> </aclSubmitApps>
        <aclAdministerApps>*</aclAdministerApps>
    </queue>
    <defaultQueueSchedulingPolicy>fair</defaultQueueSchedulingPolicy>
    <queuePlacementPolicy>
        <rule name="specified" create="true"/>
        <rule name="user" create="true"/>
    </queuePlacementPolicy>
</allocations>

So after this I stopped and started the yarn resource manager, and I see Fair Scheduler on the YARN Application console! But when attempting to run multiple jobs on cluster, the AWS EMR console shows just one job running and the other two pending. Furthermore the YARN console shows only one job running in the queue root.hadoop and don't see the other jobs (which will run after that one completes).

So how can I get the jobs to run in parallel?

Solution

  • Setting the scheduler via the yarn-site.xml does in fact work. If you pull up the YARN resource manager, the scheduler will in fact show the change, but the issue is when submitting an AWS EMR step. EMR steps inherently run sequentially, meaning AWS will not submit the next job to YARN until the previous step completes. So one had to submit the job to directly to Yarn to see the benefits; however, EMR steps seemed to have recently changed. AWS EMR now supports parallel step execution if using EMR version 5.28.0: https://aws.amazon.com/about-aws/whats-new/2019/11/amazon-emr-now-allows-you-to-run-multiple-steps-in-parallel-cancel-running-steps-and-integrate-with-aws-step-functions/.