I have a server application rewrite underway and am puzzled by the memory usage of the application. The earlier version was written with TcpListener while the new one is plain old Socket. This is mostly for performance and stability reasons which are secondary to this question and even this issue.
As mentioned, everything is heavily async'd with AcceptAsync, SendAsync, and ReceiveAsync. On top of that, I use ThreadPool.QueueUserWorkItem for utility tasks such as the initial kick-off for AcceptAsync and keeping the next AcceptAsync queued, the call after processing to write back to the Socket, and the call cleaning up disconnected clients. Further, there are a series of events that I fire with BeginInvoke and EndInvoke.
The detection for those disconnects as well as the main driver for data availability are handled by a custom class that I call AvailabilityNotifier that peaks on a ReceiveAsync as well as detecting for SocketAsyncEventArgs.BytesTransferred being zero which fires a Disconnect event.
The throughput of the application is good, and there's almost zero (relatively speaking) lock contention thanks to a healthy usage of System.Collections.Concurrent objects. However, it clings to memory like a predator clinging to a kill.
I've debugged to verify my internal collections are getting cleared, the client sockets are being shutdown and disposed of, and utilizing a buffer pool instead of creating new buffers for each read. Running a test application that ultimately performs 1,000 connections (100 concurrent) and sends/receives 100,000 messages bloats the server process memory to around 800 MB and it never goes down even after Windows clears any TIME_WAITs that might have happened. I know for sure the diposal code is firing thanks to a ton of ObjectDisposedException and null exception catch blocks that you can see in the linked github below.
I say all that without quoted code as it's quite long for a post here so here's a github: https://github.com/hoagsie/TcpServer. The Program.cs and ClientProgram.cs is provided as well if you'd want to run it yourself but the main action is in NetworkServer.cs and AvailabilityNotifier.cs. The sample server I have running also has a WCF service it talks to but is just the standard WCF project with literally no modifications. I just needed it to match a sample scenario.
I'm also not sure if it matters on some level, but I do build this in x64 mode rather than AnyCPU/x86. This is mostly for resource consumption opportunity on the target server it will be going on, but I haven't noticed a difference in behavior with regard to this issue in either x86 or x64.
EDIT:
A coworker pointed out the Snapshot tool in Visual Studio. I had never seen this before and it displayed things differently from what I had been using, which was dotTrace. It pointed to a ton of allocations around the SocketAsyncEventArgs object which makes sense, but they kept building and building. I looked at its member list again and discovered it had a Dispose method. My issue has gone away. I didn't realize that was an IDisposable object.
A coworker pointed out the Snapshot tool in Visual Studio. I had never seen this before and it displayed things differently from what I had been using, which was dotTrace. It pointed to a ton of allocations around the SocketAsyncEventArgs object which makes sense, but they kept building and building. I looked at its member list again and discovered it had a Dispose method. My issue has gone away. I didn't realize that was an IDisposable object.