merging post request in apache kafka

Question

I have 3 data sources that provide an api to access them, each provides some information every single and I want to process all the three requests and save all of them a single request in the database. So my question is, can Kafka merge these request provided all these sources have one common ID save it in DB and send a single request to a downstream system..?

Answer 1

We can store the data of different data sources in different topics and then merge the messages of multiple topics into single topic using Kafka Streams.

Scala example to merge the data

val mergedStream = streamBuilder.merge(sensor1Stream, sensor2Stream, sensor3Stream, sensor4Stream)

mergedStream.to(Serdes.String(), heartbeatSerde, "Merged-SensorsHeartbeat")

I would suggest you to follow the below link for more information.

http://www.alternatestack.com/development/app-development/kafka-streams-merging-multiple-kstream-s/

Two type of connectors are available Sink and Source Connectors. Source connector is used to write data to kafka topic from external system and Sink Connector is used to read from kafka topic and write to an external system.

Now you can read the data from merged partition using kafka connectors or debezium and write to your database.