I am working on designing a Kafka based system for message delivery from remote platforms often with poor network connectivity. The Kafka cluster would run in a cozy data center, where the failure of all the nodes is very unlikely. What worries me is the behavior of producers during the network outage (which is guaranteed to happen and can last for days). This is the main reason why the Kafka based system is not implemented yet, i.e. all data are considered critical and there should be 0 data loss. Currently we have a simple system that uses TCP connection and if unavailable, buffers the data to HDD (lets consider that there is plenty of space to buffer the data).
Does anybody have experience of using data to HDD buffering during network outages? Would this have any negative effect on the Kafka cluster? There will be hundreds of providers and it is almost guaranteed that some of them would be in outage at any given time.
Thanks for the help!
I don't see any issues with your plan. You can get this to work with Kafka if that fulfills your use case.
The Kafka Producer notifies you when a message was not sent successfully. So you'll have to handle that situation and perform the write to disk. Also when connectivity resumes, you will also have to handle sending all data that was written to disk.
None of this affect the cluster in any way.