Search code examples
amazon-web-servicesamazon-emr

EMR managed autoscaling on Step Functions

Has anyone tried creating an EMR cluster with managed autoscaling on a Step Function? Maybe I'm missing something, but in the documentation there are no mentions on how to achieve this. Is this even supported?

Solution

  • I've not tried it myself but from reading the docs it would appear to be supported. If you look at the example create cluster here it follows the standard RunJobFlow syntax. In the create cluster example defined it has the following LogUri:

    "LogUri": "s3n://aws-logs-123456789012-us-east-1/elasticmapreduce/",

    So the same level in the JSON you can add the ManagedScalingPolicy as so:

    "LogUri": "string",
"ManagedScalingPolicy": { 
  "ComputeLimits": { 
     "MaximumCapacityUnits": number,
     "MaximumCoreCapacityUnits": number,
     "MaximumOnDemandCapacityUnits": number,
     "MinimumCapacityUnits": number,
     "UnitType": "string"
  }
},

    The cluster should then be launched with a Managed Scaling policy and so long as you define instance types in your fleet config then the managed scaling should kick in, assuming you run enough workload on the cluster to trigger a scaling event.