The use case is that my consumer does some I/O job that takes a long time, occasionally. I am using Spring 2.0.1 and <spring-cloud.version>Elmhurst.RELEASE</spring-cloud.version</spring-cloud.version>
The initial solution was to set max.poll.interval.ms to a higher value and max.poll.records to a lower value than default. This did reduce the number of rebalances but there are still cases when the I/O operations take longer than max.poll.interval.ms and rebalances are quite costly, generating quite some lag. Should there be no rebalances the processing would finish in a timely manner, even considering the situations when the I/O duration spikes.
What settings and code or architecture could I use to make the system both responsive (rebalance when a consumer is actually blocked) without triggering rebalances when one consumer takes an unusually long time to process a message?
The only solution I am aware of is to tweak those two properties (or reduce the time taken by the listener).