Search code examples
phpkafka-producer-apilibrdkafka

PHPRdfKafka instead of creating new topic how to produce into existing topic


A quick question, I'm venturing into Kafka using php-rdkafka ( https://github.com/arnaud-lb/php-rdkafka ).

I went through the documentation and i can't find the syntax to produce into existing topic unless the syntax newTopic will insert to the existing topic which i doubt. I keep on getting Java error thrown and i'm not good debugging Java error. I'm seeking help from those who has been using the framework , is it a correct syntax? Please advice

<?php

$conf = new RdKafka\Conf();
$conf->set('metadata.broker.list', 'localhost:9092');

//If you need to produce exactly once and want to keep the original produce order, uncomment the line below
//$conf->set('enable.idempotence', 'true');

$producer = new RdKafka\Producer($conf);

$topic = $producer->newTopic("test"); // Is this a correct syntax to consume existing topic?

for ($i = 0; $i < 10; $i++) {
    $topic->produce(RD_KAFKA_PARTITION_UA, 0, "Message $i");
    $producer->poll(0);
}

for ($flushRetries = 0; $flushRetries < 10; $flushRetries++) {
    $result = $producer->flush(10000);
    if (RD_KAFKA_RESP_ERR_NO_ERROR === $result) {
        break;
    }
}

if (RD_KAFKA_RESP_ERR_NO_ERROR !== $result) {
    throw new \RuntimeException('Was unable to flush, messages might be lost!');
}

Solution

  • As per discussion with the developer,

    The following syntax can be used to create new topic and add the data to the following/existing topic

    $topic = $producer->newTopic("test"); /