When database is bottle neck in Kafka pair?

Question

I have Kafka stream that makes intermediate calculations then puts is to mongodb.

Problem is that mongodb can not handle a lot of inserts form Kafka stream.

How do you solve this issue?

Answer 1

I think you'll want to write the intermediate results out to a topic. Then use the MongoDB Connector to write to MongoDB. Kafka Connect is the preferred way to write to external systems from a Kafka topic. Here's the documentation for Kafka Connect