I have two Kafka consumers, subscribed to different topics and belonging to the same consumer group, having different consumer ids, running in the same thread. They are executing poll sequentially but after the first is done second seems to be stuck in poll. I tried associating them with different consumer groups and that seems to be working but unfortunately, that is not a viable solution for me.
I found this in "Kafka: The Definitive Guide":
You can’t have multiple consumers that belong to the same group in one thread and you can’t have multiple threads safely use the same consumer. One consumer per thread is the rule.
That quote directs me towards some form of thread cooperation due to the specific order of message processing I need to do.
Can someone provide an explanation of why is it necessary to run different consumers belonging to the same consumer group, subscribed to different topics in separate threads?
Thank you.
Explanation from Kafka users mailing list
Matthias J. Sax: "Why does it block? Well, after the first consumer calls poll(), it will join the group and the group will have 1 member. When the second consumer calls poll() it will try to join the group. The broker side group coordinator knows that was already one consumer in the group and it will wait until the first consumer say "I am still here and still part of the group" -- however, this will never happen, because the first consumer would do this via calling poll(), but it can't, because the second consumer is stuck in its own poll() call to actually join the group. Eventually the first consumer would time out and be remove from the group and the second consumer unblocks (the group has still one member only). Afterward the first consumer will retry to join the group... etc. etc. This ping pong game will continue forever..."