Search code examples
c++multithreadingsocketssendiocp

post many wsasend on the same socket from multiple threads

(the question is copied from msdn forum)

MSDN says this:

If you are using I/O completion ports, be aware that the order of calls made to WSASend is also the order in which the buffers are populated. WSASend should not be called on the same socket simultaneously from different threads, because it can result in an unpredictable buffer order.

Ok... but if I don't care about this order and also don't care about the order I will receive it on the other side... can I use it then? Is it safe to call WSASend from multiple threads? And will the packages be delivered how I pass them (maybe ordered differently, but complete) ??

E.g. -> I send [12a], [34b], [11c] and [223d] ... ([] marks the unit I pass to WSASend in one call) will I receive then e.g. 34b, 11c, 12a, 223d ? Or could it crash? Or could I receive things like 2a, 34b, 7a, 5c, ...

so, maybe the question should be -> is it thread safe? and is it atomic? (I'm talking about WSASend used with IOCPs, so OVERLAPPED)

Solution

  • It seems that actually calls to WSASend() are not thread safe and that if you intend to call WSASend() from multiple threads on the same connection then you should synchronise so that only one thread is actually calling into the API at any given point (i.e. hold a per connection lock around the WSASend() call). See this question and the attached test code for details: TCP/IP IOCP received data sometimes corrupt - Visual C++ on Windows and this blog entry for a more detailed explaination.

    Also note that if you are sending distinct application level messages using multiple WSASend() calls from multiple threads then you have a problem as the multiple calls could be intermingled before the data gets written to the TCP stream.

    So,

    If you have 5 threads sending 'complete 2 byte messages' such that you send

    AA, BB, CC, DD, EE etc. from 5 different threads then you will not get a stream that contains

    ABCCDDBAEE

    but you could get any combination of message order.

    If your threads are sending messages with multiple send calls, such that three sends, A1 A2 A3 from thread 1 forms a single application level message. Then all bets are off as you could end up with

    A1B1A2A3B2B3 etc. in the TCP stream.

    If you need to do the later you will either need a lock of some kind around ALL of the WSASend calls so that you can group the multiple sends into a single application level atomic send or you should use multiple WSABUFs for a single WSASend call to gather together the multiple writes into a single call.