How to sink MessagePack-encoded messages into MongoDB from Kafka

Question

I have a Kafka topic where the values are MessagePack-encoded.

Is there any way to sink the records from this topic into MongoDB using the MongoDB Kafka connector, or must the record values simply be stored as JSON?

Answer 1

You will need to find or create your own Kafka Connect Converter, then add that package to each Connect worker's classpath, followed by setting it as your key/value converter setting, from which the existing Mongo Sink Connector can deserialize the messages into a Struct and Schema form, and handle correctly.

JSON was never a requirement. Avro and Protobuf should work as well