Synchronization of multiple Kafka consumers

Question

There are two Kafka topics. Each of them has only one partition. I have a Java application with two Kafka consumers:

• first consumer for the first Kafka topic
• second consumer for the second Kafka topic

These consumers perform some changes in the database and they are synchronized using in-memory locks. Now I need to run additional instance of this application to provide high availability. But also I need to ensure that both consumers will run on the same instance of application . Otherwise I will not be able to syncronize them using in-memory locks. In general, if something bad happens with one consumer then all of them should stop on one instance and run on another.

I was trying to use @KafkaListener(topics = {topic1", topic2"}, concurrency=2) with RoundRobinAssignor. But in the end it also creates two consumers which behave independently.

Answer 1

In the end to solve this issue I've implemented custom assignor using this article https://medium.com/streamthoughts/understanding-kafka-partition-assignment-strategies-and-how-to-write-your-own-custom-assignor-ebeda1fc06f3