we are having latency issues in one of our network application. Most of the time requests are being handled within 100ms. But sometime it can take up to a few seconds for no apparent reason.
So I hooked up some monitoring tools and looked up what was happening (Wireshark to monitor the network externally through port replication and Process Monitor to see what was happening on the local machine).
I was able to match tcp packets and they usually where within a millisecond of eachother in both logs file. But in one occurence, the last packet of a series was delayed by more then 250ms in Process Monitor compared to wireshark (and the application erratic behavior - due to latency - was being observed).
Since Wireshark was hooked up on another computer I'm quite sure that what was being monitored was accurate : all the packed did reach the network card on time. As for Process Monitor I'm not totally sure about how it work : when is the network data being registered? Is it when it reach the network card? When it is made available to the application? When the application reads the data?
During these 250ms there were a few other events being registered which let me believe that Process Monitor was recording correctly and that this 250ms delay wasn't "created" by it.
Any help regarding the behavior of Process Monitor, the current method I use to dig down the problem or what you think could be the problem would be much appreciated.
Option 2
Perhaps you're experiencing the infamous 250ms delays that the GC cause from time to time (link). You can accurately measure GC suspensions using a specialized CLR host (link)
Option 1 - was ruled out
Since you are using TCP, I'd suggest that you'll turn on the NoDelay option on your socket just to eliminate the possibility that you're suffering from a clash between Nagle's Algorithm and the Delayed ACKs Algorithm. If you're experiencing "batching" of packets while sometimes a packet is "delayed" for about 200ms, then it just might be the issue.
A more in-depth explanation of this behavior can be found here.