AWS EMR provisioning fails when I use custom AMI

Question

Problem: I have an EMR cluster (along with a number of other resources) defined in a cloudformation template. I use the AWS rest api to provision my stack. It works, I can provision the stack successfully.

Then, I made one change: I specified a custom AMI for my EMR cluster. And now the EMR provisioning fails when I provision my stack.

And now my stack creation fails, due to EMR provisioning failing. The only information I can find is an error on the console: null: Error provisioning instances.. Digging into each instance, I see that the master node failed with error Status: Terminated. Last state change reason:Time out occurred during bootstrap

I have s3 logging configured for my EMR cluster, but there are no logs in the s3 bucket.

Details:

I updated my cloudformation script like so: my_stack.cfn.yaml:

rMyEmrCluster:
  Type: AWS::EMR::Cluster
  ...
  Properties:
    ...
    CustomAmiId: "ami-xxxxxx" # <-- I added this

Custom AMI details: I am adding a custom AMI because I need to encrypt the root EBS volume on all of my nodes. (This is required per documentation)

The steps I took to create my custom AMI:

1. I launched the base AMI that is used by AWS for EMR nodes: emr 5.7.0-ami-roller-27 hvm ebs (ID: ami-8a5cb8f3)
2. I created an image from my running instance
3. I created a copy of this image, with EBS root volume encryption enabled. I use the default encryption key. (I must create my own base image from a running instance, because you are not allowed to create an encrypted copy from an AMI you don't own)

I wonder if this might be a permissions issue, or perhaps my AMI is misconfigured in some way. But it would be prudent for me to find some logs first, to figure out exactly what is going wrong with node provisioning.

Answer 1

I feel stupid. I accidentally used a completely un-related AMI (a redhat 7 image) as the base image, instead of the AMI that EMR uses for it's nodes by default: emr 5.7.0-ami-roller-27 hvm ebs (ami-8a5cb8f3)

I'll leave this question and answer up in case someone else makes the same mistake.

Make sure you create your custom AMI from the correct base AMI: emr 5.7.0-ami-roller-27 hvm ebs (ami-8a5cb8f3)