EMR cluster created with CloudFormation not shown

Question

I have added an EMR cluster to a stack. After updating the stack successfully (CloudFormation), I can see the master and slave nodes in EC2 console and I can SSH into the master node. But AWS console does not show the new cluster. Even aws emr list-clusters doesn't show the cluster. I have triple checked the region and I am certain I'm looking at the right region.

Relevant CloudFormation JSON:

"Spark01EmrCluster": {
  "Type": "AWS::EMR::Cluster",
  "Properties": {
    "Name": "Spark01EmrCluster",
    "Applications": [
      {
        "Name": "Spark"
      },
      {
        "Name": "Ganglia"
      },
      {
        "Name": "Zeppelin"
      }
    ],
    "Instances": {
      "Ec2KeyName": {"Ref": "KeyName"},
      "Ec2SubnetId": {"Ref": "PublicSubnetId"},
      "MasterInstanceGroup": {
        "InstanceCount": 1,
        "InstanceType": "m4.large",
        "Name": "Master"
      },
      "CoreInstanceGroup": {
        "InstanceCount": 1,
        "InstanceType": "m4.large",
        "Name": "Core"
      }
    },
    "Configurations": [
      {
        "Classification": "spark-env",
        "Configurations": [
          {
            "Classification": "export",
            "ConfigurationProperties": {
              "PYSPARK_PYTHON": "/usr/bin/python3"
            }
          }
        ]
      }
    ],
    "BootstrapActions": [
      {
        "Name": "InstallPipPackages",
        "ScriptBootstrapAction": {
          "Path": "[S3 PATH]"
        }
      }
    ],
    "JobFlowRole": {"Ref": "Spark01InstanceProfile"},
    "ServiceRole": "MyStackEmrDefaultRole",
    "ReleaseLabel": "emr-5.13.0"
  }
}

Answer 1

The reason is missing VisibleToAllUsers property, which defaults to false. Since I'm using AWS Vault (i.e. using STS AssumeRole API to authenticate), I'm basically a different user every time, so I couldn't see the cluster. I couldn't update the stack to add VisibleToAllUsers either as I was getting Job flow ID does not exist.

The solution was to login as root user and fix things from there (I had to delete the cluster manually, but removing it from the stack template JSON and updating the stack would probably have worked if I hadn't messed things up already).

I then added the cluster back to the template (with VisibleToAllUsers set to true) and updated the stack as usual (AWS Vault).