Search code examples
amazon-web-servicesredisaws-cloudformationamazon-elasticache

Read replicas are created in wrong AZ when adding new shards through Cloudformation


I'm facing an odd error when updating my Elasticache Cloudformation Stack, at the moment it has 2 shards and every shard has a replica node, I define the AZ I want them to belong to, like this:

    "MyRedisCluster": {
            "Properties": {
                "Engine": "redis",
                "MultiAZEnabled": "true",
                "NodeGroupConfiguration": [
                    {
                        "NodeGroupId": "0001",
                        "PrimaryAvailabilityZone": "eu-west-1b",
                        "ReplicaAvailabilityZones": [
                            "eu-west-1a"
                        ]
                    },
                    {
                        "NodeGroupId": "0002",
                        "PrimaryAvailabilityZone": "eu-west-1c",
                        "ReplicaAvailabilityZones": [
                            "eu-west-1b"
                        ]
                    },
                ],
                
                "NumNodeGroups": 2,
                "ReplicasPerNodeGroup": 1,
            },
            "Type": "AWS::ElastiCache::ReplicationGroup",
            "UpdatePolicy": {
                "UseOnlineResharding": "true"
            }
        }

When I create my stack everything works correctly, my primary and replica nodes are created correctly in their appropriate AZ.

Now I want to add a new shard, so I added a new NodeGroupConfiguration object and set NumNodeGroups to 3. The update process works fine, my third shard is created, but the thing is that the primary and replica nodes are created in the wrong AZ, as you can see my third shard should have its primary node in eu-west-1a and its replica node in eu-west-1c, but they are created the other way around, primary node in eu-west-1c and replica in eu-west-1a.

{
    "NodeGroupId": "0003",
    "PrimaryAvailabilityZone": "eu-west-1a",
    "ReplicaAvailabilityZones": [
        "eu-west-1c"
    ]
}

This is soo weird, I tried to use AWS CLI to add it modify-replication-group-shard-configuration and it works just fine. Can we say this is a Cloudformation bug?


Solution

  • AWS has confirmed this is an error from their side. I created a AWS support case and also added a new issue in their cloudformation repo.

    https://github.com/aws-cloudformation/cloudformation-coverage-roadmap/issues/899