To the best of my knowledge, TCP provides at least once delivery (retransmission until an ACK), and exactly once processing at the receiver (duplicated packets are going to be just ignored and only one copy will be delivered to the app. ), if this is true why application layer messaging systems (e.g., Kafka ) and streaming systems (e.g., Spark, ) would require their own application level protocols to provide exactly once processing guarantees, why not just relying on TCP for one time delivery and/or processing?
The reliability guarantees of TCP only cover data delivery between systems and not data delivery between applications. The recipient system sends an ACK back if the data were received by the OS and put into the receive buffer of the socket. This means the ACK might be send before the application reads and processes the data. To cover guarantees about reading and processing the data by the application some kind of ACK is therefore needed inside the application protocol.