I have noticed that when I am sending packets at even intervals from a udp socket, the first packet sent seems to be delayed. For example, if I am sending the packets every 100 ms, I find the delay between receiving the packets to be normally distributed with mean 100ms and average standard deviation of 4, on my network. However, the gap between the receive time for the first and second packets is usually around 10 to 40 ms - as you can see, that's clearly a statistically significant difference, and so my question is, what's causing it?
I'm using the sendto function from C on linux. Someone suggested the delay might be caused by arp resolution preventing the packet from being sent until the destination ip has been converted to a mac address - is this likely? If I restart the sending program the first packet again takes too long though, and the delay is inconsistent - 10 to 40 ms is a pretty big range.
I need to find out why this first packet is taking too long, and how to work around it.
Edit: Further analysis with pcap indicates that the sending program is sending the packets at the right interval. Problem must be with the receiver, which is using select() to wait for a readable socket, then calling recvfrom and printing the packet. Is there some sort of buffering going on there I might not know about?
You have asked about workarounds. One possible workaround, if the reception time of the packet is important to know, is to set the SO_TIMESTAMP socket option. This will allow you to obtain the time that each packet was received by the destination kernel - you can then use this time in subsequent processing rather than the "current time".
Alternatively, you could have the sender include a high resolution timestamp in the packet that it sends, and use those.