How to seed 10 million messages into Kafka?

Question

As of my requirement on real time analytics I have to insert 10 million records into Kafka and on top of that I have to write custom consumer logic.But to fulfill this requirement how can I seed 10 million records into Kafka within 3-5 minutes time?

Preferably we want to use Spring boot as per the project requirements. Please suggest

Answer 1

Several options:

• kafka-connect-datagen, detailed in this article.

• A command line producer like kafkacat:

  seq 1 10000000 | kafkacat -b localhost:9092 -t testtopic -P
  

  • For more realistic test data use this in conjunction with Mockaroo