Search code examples
apache-kafkaspring-kafkaapache-kafka-connect

Kafka Producer vs Kafka Connector

I need to push data from database(say Oracle DB) to a kafka topic by calling a stored procedure.I need to do some validations too on message

Should i use a Producer API or Kafka Connector.

Solution

  • You should use Kafka Connect. Either that, or use the producer API and write some code that handles:

    • Scale-out
    • Failover
    • Restarts
    • Schemas
    • Serialisation
    • Transformations

    and that integrates with hundreds of other technologies in a standardised manner for ease of portability and management.

    At which point, you'll have reinvented Kafka Connect ;-)

    To learn more about Kafka Connect see this talk. To learn about the specifics of database->Kafka ingestion see this talk and this blog.