Search code examples
kubernetesapache-kafkaapache-kafka-connectconfluent-platformmongodb-kafka-connector

High availability configuration for Kafka Connect Mongodb source connector

I've been looking for specific information about high availability deployments of Kafka Connect connectors but found nothing.

In my case I have a Mongodb source connector deployed using the Confluent Helm chart. This chart supports setting the number of replicas.

Is setting replicaCount to a value >1 enough or there are other factors to consider (tasks.max, ...)?

Solution

  • If you want highly available workers, then it's pod replicas, yes.

    If you want distributed tasks across workers, that's tasks.max; if one worker dies, then tasks get rebalanced