We a Kafka producer produces messages to a topic that is consumed by only one group of consumers. The speed of consumers is quite different. We always have a certain amount of consumer lags as a buffer.
Is it possible for the producer to know the lag of each partition of the consumer group and produce messages to the partition with the lowest lag first? And is it possible if I use C/C++ clients or Java clients?
Thanks.
It is possible, given that you can lookup any consumer group name's lag from within the producer, however as the consumers are running those values will fluctuate and you're coupling yourself to knowing which consumers are running, which would be an anti-pattern in Kafka.
I'd suggest tweaking other consumer settings like poll sizes / frequency