Apache Kafka: Can we restrict message to be read by only 1 consumer?

Question

I am new to Kafka and the scenario that i am trying to achieve using Apache Kafka is:

I have a 3 producers who are sending instructions to a particular topic in Kafka. I have around 35 consumers who picks up one instruction at a time and performs the task mentioned in the instruction set. Once completed it will post the updates to an another topic in Kafka.

Requirements.

1. An instruction read by 1 consumer should not be given to another for processing
2. Message Priority * - Not a mandatory requirement
3. Only 1 instruction should be picked up at a time for processing.

I know that this can be achieved using JMS. But my organisation already has a Kafka setup ready as part of Cloud-era HDFS setup. So the using Kafka to achieve this is most preferred.

I have googled around to find a solution for this. But i was not able to find a proper answer to my 1st and most important requirement.

Any suggestions? And thanks in advance.!

Answer 1

All you have to do is to put the consumers in the same consumer group by assigning the same group.id to them. Only one instance of a consumer in the same group will get a message.

Refer to documentation at https://kafka.apache.org/documentation#intro_consumers