Search code examples
mongodbapache-kafkaapache-kafka-connectmongodb-kafka-connector

Official MongoDB Source Connector for Apache Kafka with MongoDB-4.0

I have a requirement where I have to capture MongoDB ChangeStream (inserts/updates etc) events and take some actions (save to OracleDB). I have thought of this design which seems to be good:

"MongoDB-ChangeStream" --> "MongoDB Source Connector for Apache Kafka" --> "Kafka Broker-Topic" --> "Java Service" --> OracleDB

My question here is, I am using MongoDB-4.0 and "MongoDB Source Connector for Apache Kafka" was introduced in MongoDB-4.2. Can I still use "MongoDB Source Connector for Apache Kafka" with MongoDB-4.0?

MongoDB Source Connector for Apache Kafka - Official page: https://www.confluent.io/hub/mongodb/kafka-connect-mongodb

Official page of "MongoDB Source Connector for Apache Kafka" does not talk about any prerequisite in terms of the version of MongoDB.

My second question, can this(MongoDB Source Connector for Apache Kafka) set up be done on Windows?

Appreciate any help.

Solution

  • A working solution for MongoDB(4.0) ChangeStreams with Official MongoDB Connector (mongodb-kafka-connect-mongodb-1.1.0) for Apache Kafka without using Confluent Hub Client

    1. Download Official MongoDB Kafka Connector: i. open https://www.confluent.io/hub/ ii. search "mongodb" iii. select from dropdown "MongoDB Connector for Apache Kafka" iv. Click on Download in Download installtion (note I am not using Confluent) v. unzip the downloaded file (on windows) eg. F:\software\mongodb\mongodb-kafka-connect-mongodb-1.1.0 vi. inside "etc" folder open file "MongoSourceConnector.properties" vii. Make only one change with respect to MongoDB url: connection.uri=mongodb://localhost:27017,localhost:27018 viii. One change in Kafka: In F:\software\kafka_2.12-2.4.1\config\connect-standalone.properties: plugin.path=F:\\software\\mongodb\\mongodb-kafka-connect-mongodb-1.1.0
    2. start Zookeeper: bin/zookeeper-server-start.sh config/zookeeper.properties

    3. start Kafka: .\bin\windows\kafka-server-start.bat .\config\server.properties

    4. start Kafka-Connect: bin\windows\connect-standalone.bat config\connect-standalone.properties F:\software\mongodb\mongodb-kafka-connect-mongodb-1.1.0\etc\MongoSourceConnector.properties

    5. Add new Document to MongoDB Collection and you should have a new topic created dbName.collectionName and ChangeStream event should appear on this topic.