Search code examples
javaapache-kafkakafka-producer-apiapache-kafka-connect

Read a CSV file in real time using Kafka Connect


how can I integrate Kafka connect with Kafka producer in java so that I can read a CSV file in real time? I am having a hard time finding any source related to this.

Right now I am using scala-ide to run a simple Kafka producer class, but have no idea on how to use it with Kafka connect.


Solution

  • A Kafka Connect connector for reading CSV files already exists: https://github.com/jcustenborder/kafka-connect-spooldir.

    Yuo can see an example of it in action here: https://www.confluent.io/blog/ksql-in-action-enriching-csv-events-with-data-from-rdbms-into-AWS/


    Disclaimer: I wrote the above article, and work for Confluent on whose blog it was published