Is there a Kafka counterpart to AWS Kinesis Firehose?

Question

Looks like AWS Kinesis Firehose is a way to ingest data (buffered together) to an object, is there a Kafka counterpart for this kind of functionality?

Answer 1

Yes, Kafka Connect, which is part of Apache Kafka.

You can use it with a variety of connectors, for streaming data in from a source (RDBMS, JMS, MQTT, CSV, etc etc) to a Kafka topic, and from a Kafka topic to a target (RDBMS, S3, BigQuery, HDFS, etc etc)

Kafka Connect in action:

• https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-1/
• https://www.confluent.io/blog/blogthe-simplest-useful-kafka-connect-data-pipeline-in-the-world-or-thereabouts-part-2/
• https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-3/

Disclaimer: I work for Confluent.