Why is there a delay between writing to and reading from Kafka queue?

Question

I have written a worker service to consume messages from a Kafka queue, and I have also written a test script to add messages to the queue every few seconds.

What I have noticed is that often the consumer will sit by idle for minutes at a time, while messages are being added to the queue. Then suddenly the consumer will pick up the first message, process it, then rapidly move on to the rest. So it eventually catches up, but I'm wondering why there such a delay in the first place?

Answer 1

Consumer group will take some times to contact group coordinator and get assigned partitions automatically during the delay.

If you use manual assignment, you will get less delay.