Kafka producer poor performance

Question

I have an issue with kafka producer. Actually I am using spring kafka, and sending messages through KafkaTemplate leke this:

DefaultKafkaProducerFactory<K, V> defaultKafkaProducerFactory = new DefaultKafkaProducerFactory<>(producerParams);
KafkaTemplate kafkaTemplate = new KafkaTemplate<>(defaultKafkaProducerFactory);
RecordMetadata recordMetadata = kafkaTemplate.send(record).get().getRecordMetadata();

Problem is that sometimes send message takes 4-20 sec. There a lot of messages that takes 100 ms to send. So I have few questions:

1. Is there any correlation between message size and throughput, what is the relation?

2. What should I check firstly, maybe I have't done tuning well, any direction?

Answer 1

Ok, actually problem was in machines, cpu was to high, there were logs in Cloudera manager

Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 4332ms: no GCs detected.

Detected pause in JVM or host machine (e.g. a stop the world GC, or JVM not scheduled): paused approximately 10827ms: GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=11107ms

When I run the same on machine with 8 cores - problem is gone. one more thing that was recommended is increasing Java Heap Size of Broker