i'm a little confused about the message size configuration in Kafka 2.6.0. But let's tell the story:
We are using a Kafka cluster consisting of 3 nodes. So far with the standard configuration for messages. "zstd compression" is activated.
The relevant broker configuration is simple:
compression.type=zstd
The producer configuration is also simple at this point:
compression.type=zstd
Now we want to put a messages with 8 Mbyte into a particular topic. The compressed size of this data are only 200 kbytes.
If I put this data into the topic the following error occurred:
sudo /opt/kafka/bin/kafka-console-producer.sh --topic XXX --producer.config /opt/kafka/config/admin-ssl.properties --broker-list broker < kafka/new\ 2.txt
[2020-11-05 13:43:34,500] ERROR Error when sending message to topic XXX with key: null, value: 8722456 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.RecordTooLargeException: The message is 8722544 bytes when serialized which is larger than 1048576, which is the value of the max.request.size configuration.
So i changed the producer configuration like this:
compression.type=zstd
max.request.size=10485760
Now the producer accepts larger messages. But it still does not work:
sudo /opt/kafka/bin/kafka-console-producer.sh --topic XXX --producer.config /opt/kafka/config/admin-ssl.properties --broker-list broker < kafka/new\ 2.txt
[2020-11-05 15:10:01,513] ERROR Error when sending message to topic Komsa.Kafka.Test with key: null, value: 8722544 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.RecordTooLargeException: The request included a message larger than the max message size the server will accept.
This is another error message. I don't understand why this happens.
I think this message is related to "message.max.bytes" property. But i don't unterstand why. This is the documentation for this property:
The largest record batch size allowed by Kafka (after compression if compression is enabled). If this is increased and there are consumers older than 0.10.2, the consumers' fetch size must also be increased so that they can fetch record batches this large. In the latest message format version, records are always grouped into batches for efficiency. In previous message format versions, uncompressed records are not grouped into batches and this limit only applies to a single record in that case.This can be set per topic with the topic level max.message.bytes config.
I think it means that the parameter is related to the compressed message size which are some kbyte.
Can someone help me?
I found the solution:
Problem is kafka-console-producer.sh is ignoring the compression.type in producer config. If i explicit call
sudo /opt/kafka/bin/kafka-console-producer.sh --topic XXX --producer.config /opt/kafka/config/admin-ssl.properties --compression-codec=zstd --broker-list broker < kafka/new\ 2.txt
with compression.codec=zstd it works because the producer compressed the message.