ProducerRequest send [...255, 255, 255, 255..] when the key is empty

Question

I'm using the kafka-node client and I'm trying to understand the kafka protocol. I have this basic producer example, which send message without key:

producer.send([
    { topic: "mytopic", messages: ["hi"] },
], 

And if I analyse the frame of the message (tcpdump) I get:

[
    79,   CRC
    44,
    52,
    245,  END CRC
    0,    magic_byte
    0,    partitions
    255,  ================= 
    255,
    255,
    255, ==================
    0,   size message begin 
    0,
    0,
    2,   size message end
    104,  'h'
    105,  'i'
]

So I don't understand why I have 4 bytes at 255 when the key is empty. I can't find anywhere in the documentation the explanation of this statement.

Answer 1

The key is of type bytes. Bytes uses a signed int32 to specify the length of data.

If the key is null, the length is set to -1. The representation of -1 as a signed int32 is 255,255,255,255 (0xffffffff)

It's similar for the message value. First you have the length as a signed int32, 0,0,0,2, followed by the actual 2 bytes.

[
    79,   CRC
    44,
    52,
    245,  END CRC
    0,    magic_byte
    0,    partitions
    255,  size key begin 
    255,  -1 as the key is null
    255,
    255, size key end
    0,   size message begin 
    0,
    0,
    2,   size message end
    104,  'h'
    105,  'i' ]

See this section in the Kafka Protocol Guide: http://kafka.apache.org/protocol#protocol_types

bytes, string - These types consist of a signed integer giving a length N followed by N bytes of content. A length of -1 indicates null. string uses an int16 for its size, and bytes uses an int32.