Occasionally Messages posted to the kafka went missing

Question

Hi I am having a custom framework, which exposes API to publish as well as consume. I support a lot of topics for a user to post to and consume. This seems to work fine except that occasionally, after I publish the message to the topic, my consumer does not get any message. It is weird because kafka is up and running fine. Other messages to other topics before and after this missed messages work fine. Messages to the same topic is also fed to the consumer after this incident.

If it helps, I have only one partition per topic. When I open the KafaServerLog/data directory and search for the specific topic, in my development environment, I see entries everytime, I publish and consume. However the issue where this occurs( in a client place) the log files are empty. Am I missing something here?

Answer 1

Producer api gives you a callback which you can use to trace what is the occasional problem in producer. A sample code will look something like this:

try (Producer<String, String> producer = new KafkaProducer<>(producerProps)) {
    producer.send(new ProducerRecord<String, String>("My-topic", "message payload"), new Callback() {
        public void onCompletion(RecordMetadata metadata, Exception ex) {
            if (ex != null) {
                LOGGER.warn(String.format("Failed to produce record. Got Exception: %s", ex));
            } else {
                LOGGER.info("Sent record successfully");
            }             
        }
    });
}