I have a service, which is running in Kubernetes, During peak time, this service can scale in 1000s. Each service instance is consuming events from Kafka, We are using the Managed Streaming Kafka(AWS MSK) as broker, Since topics are going to be so many, and as topic has its metadata, which would leads to performance overhead, so I was thinking of a single topic with multiple partitions and each service instance attached to specific partition. Our backend is in typescript, and we are using kafkajs, but it seems library doesn't support custom assignment strategy, where as consumers in a consumer group keeps on increasing, we keep on attaching them on partitions sequentially. Does any one know, any kafka library in javascript which supports this specific feature.
KafkaJS does support individual partition assignment via partitionAssigners consumer option.
You can use a StatefulSet in Kubernetes with an emptyDir volume for stateless apps, to predict what instance id a given pod will have, then use that to target a specific partition.
However, you cannot have more replicas than partitions, anyway. So unless you have thousands of partitions to consume, you're wasting Kubernetes resources by scaling that much. You should look into KEDA autoscaler rather than using CPU/MEM load, then treat every partition the exact same, and not associate partition value to your process orchestration.