Search code examples
socketsnetwork-programmingprotocolscommunication

TCP communication protocols for realtime data plus metadata

I'm fairly new to network protocol design and would love suggestions to my problem.

My Requirements

  1. I need to stream realtime data from a TCP socket client to a server. I have no choice of UDP here.
  2. The data needs metadata descriptors to be correctly decoded on the server side. This descriptors can change at run time on behalf of client user.
  3. I need to send occasional control commands from client to server for data operations.

Design A

  • Set up two pairs of socket server and client, one for Requirement #1 and the other for Requirement #2 and #3; call them Data Tunnel, and Metadata Tunnel for now.

Pros

  • Clear in concept and easy to isolate metadata and data modules in the implementation.
  • Realtime data, i.e., #1, could be sent as bytes and needs no extra data encapsulation.

Cons

  • Synchronization between both tunnels is needed to avoid decoding error or glitches due to control changes.
  • May need two threads per socket-client pair to avoid blocking the main thread of the applications: one thread for each tunnel.

Design B

  • Use one tunnel only.
  • Design at least two packet data structures for realtime data and metadata, respectively. The realtime data packet will include its interpretation.
  • Let data and metadata share the only tunnel.

Pros

  • Each realtime-data packet is bundled with metadata and will be guaranteed to decode correctly.
  • Uses only one thread.

Cons

  • More complex implementation on receiver data handling, e.g., must distinguish data from metadata on receiving a packet.
  • Busier traffic on the single tunnel. I'm betting on control messages won't clog realtime data consumption, but I don't really know about that.

I'm leaning towards Design A to be more safe on the "realtime" guarantee than on correct decoding. But I want to double-check my ideas with experts.

I know there are hundreds of communication protocols out there, but to be honest, it makes me dizzy when looking at random protocol documentation. My application is relatively lightweight after all, so rolling my own seems to make more sense.

By the way, I use Google Protobuf for protocol design. I assume its performance is realtime-ready.

Any suggestions would be greatly appreciated.

Solution

  • As @RonMaupin mentioned in the comments, TCP and "real time" aren't particularly compatible, since TCP prioritizes "getting the bytes there correctly" over "getting the bytes there in a particular time frame".

    That said, what you can achieve with TCP is "getting the bytes there as quickly as possible", as long as you are okay with the fact that in some circumstances (e.g. a network that is dropping lots of packets), "as quickly as possible" might not be all that quickly.

    Regarding how many TCP streams to use, the quality of TCP that is relevant to making that decision is that a TCP stream always enforces in-order/FIFO delivery of the bytes in that stream. That is, all the bytes you send() to a TCP socket will be recv()'d in that exact order by the receiving program -- that's a behavior that can work for you, or against you, so you want to design your program in such a way that it works for you. In particular, when deciding whether to use one or multiple TCP connections, ask yourself, "does the data I'm sending need to be received in the same order it was sent, or does that not matter?" If strict FIFO ordering is important, than a single TCP stream is the way to go; OTOH if you e.g. have two types of data and the type-B data is logically independent of the type-A data, then you might consider giving the type-B data its own separate TCP stream, so that a dropped-packet-of-A-data wouldn't slow down the transmission of B-data.

    In any case, you'll want to have at least some minimal protocol/framing (e.g. at least message-type and message-size header-fields before each data-message) so that the receiver doesn't have to guess at the meaning of the bytes it is receiving. (even if you don't need them at first, you will want them in your second release as a way to help maintain backwards-compatibility with previous versions of the protocol)

    Some other suggestions for making your TCP-data as fast/low-latency as possible:

    1. Disable Nagle's algorithm (either permanently or at least momentarily after you've finished send()-ing a particular burst of data) -- otherwise you'll get 200+ milliseconds of unnecessary latency most of the time.

    2. Assuming your programs run on platforms with plenty of RAM, setsockopt() with the SO_SNDBUF and SO_RCVBUF options to make your sending and receiving socket-buffers as large as possible; that reduce the possibility of the buffers filling up and packets getting dropped due to no-space-available-in-a-buffer.

    3. If possible, design your sending algorithm to generate the data-to-send only at the last possible moment, rather than queueing up large amounts of data-to-be-sent in advance. For example, if (due to some trigger-event) your code decides that it needs to send the current state of a particular data-structure across the TCP socket ASAP, rather than serializing the data structure and enqueueing (and/or send()-ing) the serialized bytes right away, just set a dirty-flag indicating that the structure needs to be sent. Then, the next time the socket indicates that it is ready-for-write, that is the time to serialize the data structure and send it to the socket. The benefit is that if you receive e.g. 10 trigger-events in quick succession, with this dirty-flag design you still end up only sending the final version data structure across one time, rather than sending it 10 times in a row. A secondary benefit is that this limits the backlog of data that can queue up waiting to be sent, and thereby reduces the average latency of the data-updates.

    4. On the receiving side, have the recv() calls done in a tight loop by a dedicated, high-priority thread, which does very little other than receive data as quickly as possible and then enqueue it for further processing. The idea here is to minimize the possibility of the receiving TCP socket's incoming-data-buffer ever becoming full, because if it does become full, some incoming TCP packets may get dropped, forcing a TCP backoff-and-resend, which will slow down transmission.