I need to use the consume-process-produce pattern for processing Kafka messages and have configured a Spring Kafka listener container with a Kafka Transaction Manager and have also set the transaction-id-prefix to enable Kafka transactions. I am using an ack-mode of BATCH and trying to understand when the offsets actually get committed in this mode with transactions. The documentation seems to indicate that ack-mode BATCH commits offsets once the all the records from a poll have been consumed - Is this true in the transactional context too i,e 1 transaction per poll?
OR is that ack-mode BATCH ignored when using Kafka Transactions (1 transaction per listener invocation)? Is so, how adversely does it impact the consumer performance given that the consumer would need to talk to broker for every single record read? Does this also mean that my response producer can no longer buffer records and send them to broker in batches and i lose producer asynchronicity?
Sorry to ask multiple queries in the same post (perhaps too many).
The AckMode is not used at all when transactions are being used. The offset for the record is sent when the listener exits and before the transaction is committed.
You can improve things by using a batch listener where you get all the records in a List<> in a single call.
However, in order to properly support producer fencing, you should also set subBatchPerPartition (added in 2.3.2) to true so we get a transaction for each topic/partition returned by the poll().
Again, the offsets for the sub-batch will be sent when the listener exits.
If you don't care about fencing, you can process the entire batch in one transaction.