I have an application (in Java 8) that sends messages to Kafka. In order to improve the performance I want to configure the producer to send messages in batches (using ProducerConfig.BATCH_SIZE_CONFIG and ProducerConfig.LINGER_MS_CONFIG). But now, if I understand correctly, when for some reason the program crashes before sending the buffer, I will lose the messages.
In order to avoid this problem I would like to have to have some kind of WAL on the producer side. Is there any ready-made solution (program, library, Kafka configuration) that gives something like this? I would prefer not to implement it myself.
There's no builtin solution for this, no. Therefore you'll need to implement this yourself, or add a shutdown hook to your application to flush the producer