Search code examples
javaapache-kafkapubnub

Can't Produce PubNub data stream to Kafka Cluster

I am currently trying to pass live streaming data from PubNub to a Kafka cluster. I am able to get the data from PubNub but when I try to produce to Kafka, I get the following error:

Exception in thread "Subscription Manager Consumer Thread" java.lang.NullPointerException
    at org.apache.kafka.clients.producer.KafkaProducer.propsToMap(KafkaProducer.java:1238)
    at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:298)

This occurs as soon as the Kafka cluster is initialized. I've looked around but there isn't much information on this.

Any help is greatly appreciated.

Thank you

Solution

  • Produce PubNub data stream to Kafka Cluster

    There is an official approach to bridging Kafka Topics to PubNub Channels. Using Docker.

    Here is the test example that should get you up and running in 60 seconds. Start the docker compose file in a terminal window. This will launch Kafka, Zookeeper and a sample feed generator on the topic topic.

    For security, you will need to get your private API keys from: https://dashboard.pubnub.com/signup The following API Keys are for public-use and may be rotated.

    Open a new terminal session and run the following commands:

    In the following command, notice KAFKA_TOPIC=topic is the Kafka Destination Topic and PUBNUB_CHANNEL=channel is the PubNub Source Channel.

    cd kafka-bridge
docker build -f kafka/plain/dockerfile -t kafka-bridge .
docker run                                                                        \
    --network=host                                                                \
    ## ~ Replace with your own API Keys ~ https://dashboard.pubnub.com/signup     \
    -e PUBNUB_PUBLISH_KEY=pub-c-6b57a39e-79e7-4d1d-926e-5c376a4cb021              \
    -e PUBNUB_SUBSCRIBE_KEY=sub-c-df3799ee-704b-11e9-8724-8269f6864ada            \
    -e PUBNUB_SECRET_KEY=sec-c-YWY3NzE0NTYtZTBkMS00YjJjLTgxZDQtN2YzOTY0NWNkNGVk   \
    ## ~ Replace with your own API Keys ~ https://dashboard.pubnub.com/signup     \
    -e PUBNUB_CHANNEL_ROOT=''                                                     \
    -e PUBNUB_CHANNEL='channel'                                                   \
    -e KAFKA_GROUP=test-group                                                     \
    -e KAFKA_TOPIC=topic                                                          \
    -e KAFKA_BROKERS=0.0.0.0:9094                                                 \
    kafka-bridge

    PubNub Kafka