kafka - message with partition key put into wrong partition

Question

I would like to know, under which situation does messages with same partition key go into different partition.

I had run two consumers belonging to the same group listening on a topic in console using the command given below:

sudo /etc/kafka/kafka_2.11-1.1.0/bin/kafka-console-consumer.sh --
bootstrap-server localhost:9092 --topic demo3  --consumer.config 
config/consumer.properties --property print.key=true --property 
key.separator=:

I used "nmred/kafka-php" library to put messages to the topic demo3 with key abc. When i send multiple such messages, i found that few of the messages goes to the second consumer whereas most of the messages goes to consumer 1.

As i used the same key abc for all messages, i expected all the messages to be consumed by the same consumer. Each consumer is binded to each partition.

I use the following code to produce messages:

$config = \Kafka\ProducerConfig::getInstance();
$config->setMetadataRefreshIntervalMs(10000);
$config->setMetadataBrokerList('x.x.x.x:9092', 'y.y.y.y:9092');

$config->setRequiredAck(1);
$config->setIsAsyn(false);
$config->setProduceInterval(500);
$producer = new \Kafka\Producer(
function() {
    return [
        [
            'topic' => 'demo3',
            'value' => 'test message.',
            'key' => 'abc',
        ],
    ];
}
);

$producer->success(function($result) {
 var_dump($result);
});
$producer->error(function($errorCode) {
    var_dump($errorCode);
});
$producer->send(true);

This screenshot clearly shows that 3 messages had been sent to one consumer and the other one to another consumer

Answer 1

What you are saying is true, you should see messages with same key "abc" being consumed by the same consumer. Can you check if a rebalancing starts and maybe the first consumer left partition to the other ? Or, using this php kafka producer (I have never used it) can you trace in which partition each message is put. A kafka producer should get the RecordMetadata information when sending a message in order to know at which partition and offset the message is assigned. Because it's the producer to decide the destination partition, you should be sure the the php kafka producer is working fine.