Search code examples
mongodbapache-kafkaconfluent-platformkafka-topic

Kafka topic returns 0 messages


I have a real-time process that streams events occuring in MongoDB directly to Snowflake using Confluent Platform. Confluent is established on my local machine as a docker image and I see no issues:

     Name                    Command                  State                         Ports
------------------------------------------------------------------------------------------------------------
broker            /etc/confluent/docker/run        Up             0.0.0.0:9092->9092/tcp
connect           /etc/confluent/docker/run        Up (healthy)   0.0.0.0:8083->8083/tcp, 9092/tcp
control-center    /etc/confluent/docker/run        Up             0.0.0.0:9021->9021/tcp
ksql-datagen      bash -c echo Waiting for K ...   Up
ksqldb-cli        /bin/sh                          Up
ksqldb-server     /etc/confluent/docker/run        Up (healthy)   0.0.0.0:8088->8088/tcp
rest-proxy        /etc/confluent/docker/run        Up             0.0.0.0:8082->8082/tcp
schema-registry   /etc/confluent/docker/run        Up             0.0.0.0:8081->8081/tcp
zookeeper         /etc/confluent/docker/run        Up             0.0.0.0:2181->2181/tcp, 2888/tcp, 3888/tcp

Everything was working over last days and today I re-ran my process with powershell:

docker-compose exec broker bash and kafka-console-consumer --topic <MY_TOPIC> --bootstrap-server broker:9092

I noticed that when I insert data to Mongo's collection Kafka topic doesn't see it, instead returns Processed a total of 0 messages

How to debug what is going on? Connection passwords, topics are all set correctly and I don't know where to start debugging.

I used to get below result but now it's idle: enter image description here


Solution

  • Ok, in Confluent Control Center Connect tab check setting for source connector.

    For my mongo's connection, new URI syntax fixed the issue (the one where you don't point database, that is done in seperate tab)

    enter image description here