What is a proven lightweight application level way to guarantee the delivery of messages? I have done some reading on using sequence numbers and acknowledging the receipt of these, but there might be better ways? The transport is currently tcp, but that might change.
I can give you an example how it works for XMPP protocol.
Mobile apps use TCP as a transport for it, and there also could be some cases where messages are lost (e.g. issues with connectivity etc.)
So they came up with a separate XEP called Stream Management https://xmpp.org/extensions/xep-0198.html
Both parties exchange with additional simple packages: r and a.
The full flow looks like the following:
<!-- User A sends a message to User B -->
<message from='[email protected]/churchyard'
to='[email protected]'
xml:lang='en'>
<body>Hello</body>
</message>
<!-- then User A requests an acknowledgment from User B whether he received a message or not -->
<r xmlns='urn:xmpp:sm:3'/>
<!-- User B answeres with an acknowledgment package and a count how many packages he received -->
<a xmlns='urn:xmpp:sm:3' h='1'/>
So it's not any magical stuff here - it's just an additional data is exchanged to make sure it's all delivered properly. I guess you can follow a similar way.