AWS EMRFS Consistent View enable via SDK

Question

Typically emrfs consistency is enabled via emrfs-site.xml http://docs.aws.amazon.com/ElasticMapReduce/latest/DeveloperGuide/emrfs-configure-consistent-view.html

Does anyone know if these setting can be accessed via the SDK?

Answer 1

To enable EMRFS with the Java SDK, an "emrfs-site" configuration needs to be added to the RunJobFlowRequest and the fs.s3.consistent property must be set to true. Like this:

Map<String, String> emrfsProperties = new HashMap<>();
emrfsProperties.put("fs.s3.consistent", "true");

RunJobFlowRequest request = new RunJobFlowRequest()
        ....
        .withServiceRole(SERVICE_ROLE)
        .withJobFlowRole(JOB_FLOW_ROLE)
        .withConfigurations(
                new Configuration().withClassification("yarn-site").withProperties(yarnProperties),
                new Configuration().withClassification("emrfs-site").withProperties(emrfsProperties)
        )
        .withInstances(new JobFlowInstancesConfig()
                .withEc2KeyName(EC2_KEYPAIR)
                ....

A full list of EMRFS configuration parameters can be found here