Search code examples
amazon-s3apache-kafkaapache-kafka-connects3-kafka-connector

Kafka S3 Sink basic doubts

Do I really need to use confluent (CLI maybe)? Can I write my custom connector?
How can I write my first Kafka Sink? How to deploy them?

For now, let's assume we have the following details:

Topic: curious.topic
S3 bucket name: curious.s3
Data in the topic: Text/String
My OS: Mac
Solution

  • You start at the documentation for S3 Sink, looking over the configuration properties, and understand how to run Connect itself and deploy any connector (use the REST API); no, confluent CLI is never needed.

    You don't need to "write your own sink" because Confluent already has an S3 Sink Connector. Sure, you could fork their open-source repo, and compile it yourself, but that doesn't seem to be what you're asking.

    You can download the connector using different command confluent-hub.

    Note: pinterest/secor does the same thing, without Kafka Connect.