We're working with a latency-critical application at 30fps, with multiple stages in the pipeline (e.g. compression, network send, image processing, 3D calculations, texture sharing, etc).
Ordinarily we could achieve these multiple stages like so:
[Process 1][Process 2][Process 3]
---------------time------------->
However, if we can stack these processes, then it is possible that as [Process 1] is working on the data, it is continuously passing its result to [Process 2]. This is similar to how iostream works in c++, i.e. "streaming". With threading, this can result in reduced latency:
[Process 1]
[Process 2]
[Process 3]
<------time------->
Let's presume that [Process 2] is our a UDP communication (i.e. [Process 1] is on Computer A and [Process 3] is on Computer B).
The output of [Process 1] is approximately 3 MB (i.e. typically > 300 jumbo packets at 9 KB each), therefore we can presume that when we call ZeroMQ's:
socket->send(message); // message size is 3 MB
Then somewhere in the library or OS, the data is being split into packets which are sent in sequence. This function presumes that the message is already fully formed.
Is there a way (e.g. API) for parts of the message to be 'under construction' or 'constructed on demand' when sending large data over UDP? And would this also be possible on the receiving side (i.e. be allowed to act on the beginning of the message, as the remainder of the message is still incoming). Or.. is the only way to manually split the data into smaller chunks ourselves?
Note:
the network connection is a straight wire GigE connection between Computers A and B.
No, you can't realistically do it. The API doesn't provide for it, and ZeroMQ promises that a receiver will get a complete message (including multi-part messages) or no message at all, which means that it won't present a message to a receiver until it's fully transferred. Splitting the data yourself into individually actionable chunks that are sent as separate ZeroMQ messages would be the right thing to do, here.