Why is the Completed callback from SocketAsyncEventArgs frequently executed in newly created threads instead of using a bounded thread pool?

Question

I have a simple client application that receives byte buffers from the network with a low throughput. Here is the code:

private static readonly HashSet<int> _capturedThreadIds = new HashSet<int>();

private static void RunClient(Socket socket)
{
    var e = new SocketAsyncEventArgs();
    e.SetBuffer(new byte[10000], 0, 10000);
    e.Completed += SocketAsyncEventsArgsCompleted;

    Receive(socket, e);
}

private static void Receive(Socket socket, SocketAsyncEventArgs e)
{
    var isAsynchronous = socket.ReceiveAsync(e);
    if (!isAsynchronous)
        SocketAsyncEventsArgsCompleted(socket, e);
}

private static void SocketAsyncEventsArgsCompleted(object sender, SocketAsyncEventArgs e)
{
    if (e.LastOperation != SocketAsyncOperation.Receive || e.SocketError != SocketError.Success || e.BytesTransferred <= 0)
    {
        Console.WriteLine("Operation: {0}, Error: {1}, BytesTransferred: {2}", e.LastOperation, e.SocketError, e.BytesTransferred);
        return;
    }

    var thread = Thread.CurrentThread;
    if (_capturedThreadIds.Add(thread.ManagedThreadId))
        Console.WriteLine("New thread, ManagedId: " + thread.ManagedThreadId + ", NativeId: " + GetCurrentThreadId());

    //Console.WriteLine(e.BytesTransferred);

    Receive((Socket)sender, e);
}

The threading behavior of the application is quite curious:

1. The SocketAsyncEventsArgsCompleted method is frequently run in new threads. I would have expected that after some time no new thread would be created. I would have expected the threads to be reused, because of the thread pool (or IOCP thread pool) and because the throughput is very stable.
2. The number of threads stays low, but I can see in the process explorer that threads are frequently created and destroyed. Likewise, I would not have expected threads to be created or destroyed.

Can you explain the application behavior?

Edit: The "low" throughput is 20 messages per second (roughly 200 KB/s). If I increase the throughput to more than 1000 messages per second (50 MB/s), the application behavior does not change.

Answer 1

The low application throughput itself cannot explain the thread creation and destruction. The socket receives 20 messages per seconds, which is more than enough to keep a thread alive (the waiting threads are being destroyed after spending 10 seconds idle).

This problem is related to the thread pool thread injection, i.e. the threads creation and destruction strategy. Thread pool threads are regularly injected and destroyed in order to measure the impact of new threads on the thread pool throughput.

This is called thread probing. It is clearly explained in the Channel 9 video CLR 4 - Inside the Thread Pool (jump to 26:30).

It seems like thread probing is always done with newly created threads instead of moving a thread in and out of the pool. I suppose it works better like this for most applications because it avoids to keep an unused thread alive.