Search code examples
apache-kafkaapache-kafka-streams

Can Kafka Streams read the data from s3 and send them to other destination such as mysql or olap?


There are little articles about kafka streams which reads the data from s3 and send them to other destination such as olap or oltp.

I cannot see the example code of them so am I correct that kafka streams cannot read s3 and then send them to database systems?


Solution

  • Kafka Streams must read from a Kafka topic.

    There is an S3 Source Kafka Connect plugin to get data into a Kafka topic, which you can then use Kafka Streams to consume

    Similarly, you should use Kafka Connect Sink plugins to write to a database. Kafka Streams is not designed to be used that way.