I want to write a bash script to constantly add new files to a folder and use cat to send the file contents to a Kafka producer with:
cat $FILEPATH |$KAFKA_HOME/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test
But it generally takes more than 4 seconds to finish the execution. I am wondering if there's any way to write a script to automatically add new messages to a Kafka producer with less delay.
I also tried to echo the contents of the file to the producer instead, but the delay still exists. So, I think it has something to do with the producer command.
The JVM needs to startup, yes, but that shouldn't take very long. You could try installing kcat command instead, but you should first try without using cat
kafka-console-producer ... < $FILEPATH
Otherwise, you're running the producer command for each line of the file Useless use of cat?
automatically add new messages to a Kafka producer
This doesn't require a shell script. You can use Kafka Connect, Filebeat, Fluentbit, etc to watch a directory of files automatically and produce their lines to Kafka