I am trying to implement a Kafka consumer using Kotlin co-routines. I have placed the consumer poll() function inside the co-routine. For reading from multiple partitions, I have run two consumers subscribed to same topic with same group-id. The actual problem is that, the consumer poll() is turning into a long running function and hence utilizing all the worker threads.
The issue is only when I run two consumers with same group-id. When I run consumers with different group-id, there won't be such issue.
Don't use coroutines with poll. Coroutines should not used with blocking methods that take relatively long, and poll is such a method. I'm not sure there is a non-blocking poll available in the Java client, if it is, you should use that instead.