MSK Kafka: Please explain why there are multiple endpoints listed for ZookeeperConnectString and how I should be using them

If you run the following aws command, you will get msk kafka cluster details:

aws kafka list-clusters

This returns something like this:

{
    "ClusterInfoList": [
        {
            "BrokerNodeGroupInfo": { ...},
            "ClusterArn": "arn:aws:kafka:us-west-2:000000000000000:cluster/my-cluster/{guid}",
            "ClusterName": "my-cluster",
            "ZookeeperConnectString": "z-1.my-cluster.xyz.c5.kafka.us-west-2.amazonaws.com:2181,z-3.my-cluster.xyz.c5.kafka.us-west-2.amazonaws.com,z-3.my-cluster.xyz.c5.kafka.us-west-2.amazonaws.com"
        }
    ]
}

Notice how the ZookeperConnectString is a comma separated string...

Is it really zookeeper (I thought the whole point of zookeeper was to coordinate the brokers with one single endpoint) or is it a list of brokers? That's what I suspect. Also, when I use this in the -zoookeeper command to create topics or to produce to a topic, it works just the same if I just use one of the endpoints.

Can I get away with using just one of the endpoints in my application? Or should I use all three in my producer config. If so, why? What are the repercussions of using just one of them?

Solution

Those are not the brokers, but the different zookeeper servers that form the zookeeper ensemble for your Kafka cluster.
You can use just one of them, but that implies the specific zookeeper must be running in order for the command to succeed.
You should use all of them in order to achieve high-avaliability and fault-tolerance at the start of your clients, avoiding the scenario in which the zookeeper you just set on your configs is stopped (while the others are still running).

Setting all of them guarantees (if the quorum is healthy) that your kafka command will be successful even if some of the zookeeper servers are not alive.

Zookeeper Multi Server Setup

For reliable ZooKeeper service, you should deploy ZooKeeper in a cluster known as an ensemble. As long as a majority of the ensemble are up, the service will be available. Because Zookeeper requires a majority, it is best to use an odd number of machines. For example, with four machines ZooKeeper can only handle the failure of a single machine; if two machines fail, the remaining two machines do not constitute a majority. However, with five machines ZooKeeper can handle the failure of two machines.