How to reduce latency between kafka's producer and consumer?

Question

I am using kafka with spring boot. I read so many articles on kafka tuning and made some configuration changes on both producer and consumer sides.

But still kafka took 1.10 seconds to reach one message from producer to consumer. As per kafka, kafka is very well programmed for high latency but I am not able achieve.

ProducerConfig :

@Bean
    public Map<String, Object> producerConfigs() {
        Map<String, Object> props = new HashMap<>();
        props.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, StringSerializer.class);
        props.put(ProducerConfig.BATCH_SIZE_CONFIG, 20000);
        props.put(ProducerConfig.LINGER_MS_CONFIG, 1000);
        return props;
    }

ConsumerConfig

@Bean
    public Map<String, Object> consumerConfigs() {
        Map<String, Object> props = new HashMap<>();
        props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
        props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, StringDeserializer.class);
        props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
        props.put(ConsumerConfig.RECEIVE_BUFFER_CONFIG, 502400);
        return props;
    }

Some Points which I want to described :

1. Producer : for low latency, Batch.size and liner.ms is very crucial on producer configuration. Try to set as maximum.
2. Consumer : For high-throughput, need to increase buffer size.

I had these two changes but still not able to achieve what I am expected from Kafka.

1. How will be achieve High latency using kafka ? Any specific configuration in producer and consumer ?

Any suggestions will help me .

Answer 1

There is a nice white paper on Optimizing your Kafka Deployment by Confluent which explains in full details how to optimize the configurations for

• latency
• throughput
• durability
• availability

You are interested in reducing latency. Your current configuration can lead to a high latency because linger.ms is set to 1000 (1 second). In addition, the batch.size is "quite high" which probably leads to a delay of at least 1 second (from linger.ms).

To reduce your latency, you can force your producer to send messages without any delay and irrespective of their size by setting linger.ms=0.

Overall, the white paper give the following recommendation, which is also what I have experienced in practice and can highly support:

Producer:

• linger.ms=0
• compression.type=none
• acks=1

Consumer:

• fetch.min.bytes=1