How does RD_KAFKA_PARTITION_UA work in librdkafka?

Question

I have a producer in php which publishes to queue. I am using php-rdkafka library for the same.

Following is the code:

        $conf->set('log_level', LOG_DEBUG);
        $conf->set('debug', 'all');
        $rk = new RdKafka\Producer($conf);
        $rk->addBrokers("34.93.118.165:9092");
        $topic = $rk->newTopic("download_first");
        $topic->produce(RD_KAFKA_PARTITION_UA, 0, utf8_encode($message));
        $rk->flush(100);

and I have 2 python consumers consuming from the queue.

The topic has 10 partitions. Most of the times when a new message is produced it goes to the same partition as before which is being consumed by a single consumer and as a result other consumer sits idle.

Is there any other partitioner assigning thing I can use while producing messages?

Answer 1

https://github.com/arnaud-lb/php-rdkafka uses librdkafka underneath which is a C based Kafka library.

As per the docs RD_KAFKA_PARTITION_UA is for using automatic partitioning using the topic's partitioner function or you can pass the exact partition number instead of RD_KAFKA_PARTITION_UA to force a partition.

Or you call $conf->setPartitioner(RD_KAFKA_MSG_PARTITIONER_RANDOM); for enabling random partitioner function. Here's the list of possible constants for setPartitioner