A typical server application would create a socket for each incoming connection spawn a new thread.
However is it possible to do the demuxing yourself in a single thread? What I want is that the application keeps a state machine for each end point which deserializes message as data becomes available in a single thread.
Threading is the simplest, and I have a working implementation of that. However each thread adds overhead fairly quickly, and lingering connections will require resources. What I want is that instead for each connection a state machine builds up a message (which the threaded code already does anyway) and upon completion dispatches the deserialized message to a worker queue.
Is this possible in .NET? P/Invoke would also be an acceptable solution.
A typical server application would create a socket for each incoming connection spawn a new thread.
That is not a true statement. Very few servers — only those which need not scale to large numbers of connected clients, and not even all of those — will dedicate an entire thread to a single connection.
I would say only the most rudimentary servers would use a thread-per-connection design. I suppose that since there are a lot more experimental servers (i.e. hobbyists learning to write network code) than production servers, the sheer numbers might be on the side of thread-per-connection.
But IMHO only production servers are really relevant to the question, as they show what good design and implementation would be. And for those, thread-per-connection is definitely going to be in the minority.
However is it possible to do the demuxing yourself in a single thread?
Using Socket.Select() one can have a single thread dedicated to the handling of several sockets.
However, more typical would be to use one of the several asynchronous programming APIs available for use with sockets, which uses I/O completion ports to farm out handling of I/O operations to a pool of threads dedicated for the purpose. This allows concurrency but efficient use of the threads to minimize context-switching.
My preferred approach with the current .NET 4.5 and C# 5.0 features is to wrap the socket in an instance of NetworkStream, so that I can use the ReadAsync() and WriteAsync() methods, which in turn allow the use of await in the C# code. This makes the asynchronous code a lot easier to read and implement.
But you can use e.g. Socket.BeginReceive() or Socket.ReceiveAsync() (the latter being useful for servers that require extremely high scalability…even the former is still much more scalable than thread-per-connection though) just as well.
Regardless of API used, the typical server implementation will include a state object class for each connection. This can be done with thread-per-connection design as easily as with the async I/O designs. Your state machine would reside in that state object class, so that the processing of I/O operations on the socket can use the state machine.