I'm running a project. My goal is to design a Kafka cluster. Preventing the condition that when one machine die, the data on that Kafka all disappear. I wonder if Kafka replication can be set on different machine in order to back-up?
In addition to setting a replication factor of 2 or 3 on your topics to ensure backup replica copies are eventually created, you should also publish messages with acks=all to ensure that acknowledgements indicate a guarantee that the data has been written to all the replicas. Otherwise with acks=1 you get an ack after only 1 copy is committed, and with acks=0 you get no acks at all so you would never know if your published messages ever made it into the Kafka commit log or not.
Also set unclean leader election parameter to false to ensure that only insync replicas can ever become the leader.