Search code examples
amazon-web-servicesapache-kafkaamazon-ecsapache-kafka-connect

Kafka connect cluster architecture in aws ecs

I have a KAFKA CONNECT in an ECS SERVICE with a single task. It has an ASG and an ALB.

I know that when I start my ECS TASK with my KAFKA CONNECT, I generate 1 WORKER from the KAFKA CONNECT. In this WORKER I can have several TASKS tasks.max created in the CONNECTORS CONF.

My reasoning is as follows: by scaling my ECS SERVICE to 3 ECS TASKS, I would have 3 WORKERS from KAFKA CONNECT and by scaling my TASKS tasks.max to 3 I would arrive at the logic of the image?

ecs cluster

If this is true, do I need any additional configuration so that the cluster is managed and the PRODUCER data reaches the TOPIC and is received by the CONSUMER without duplication?

Solution

  • Workers are the replicas of your container, yes.

    Tasks can be more than workers, and are generally limited by available CPU cores or threads, but the JVM manages that logic.