on a project I am working on, we are seeing out-of-order issues in certain circumstances on a SMP system when we are reading a UDP stream from the network. We can see it arrives from the network in order by sniffing off a hub connected between the sender and receiver. However sometimes it appears to arrive out of order when read from the socket. Is there any guarantee for UDP packets in this case or should the application implement a re-order buffer? We are not setting the CPU affinity here, I suspect that might help but ideally I would like all the CPU/hw threads to handle the network traffic.
UDP does not guarantee any ordering whatsoever. It is the responsibility of the application. In fact it does not even guarantee that the packets won't be repeated/ dropped etc. I suggest you read: http://en.wikipedia.org/wiki/User_Datagram_Protocol
It does not make sense for the kernel to make any such guarantees, especially if the incoming packets themselves could be out of order, as the kernel can (reasonably) expect that the application will deal with it, if the application requires ordering.