WCF + NetTcp: high load make the channel stop working (calls/second rate)

First of all, sorry, i'm not fluent.

I'm trying to figure out why my WCF services stop working when we have an environment with high calls/second rate. I'm not sure that just increasing timeout will solve the issue.

We have 2 webservices:

The first is hosted on IIS 7.5, Windows Server 2008 R2 Enterprise SP1 x64, with AppFabric (and WAS)
Second, hosted on Windows Service, Windows 2003 R2 SP1 x86

Both webservices have minimum configuration: No authentication, No trasaction, Without special treating of message.. check the binding:

<netTcpBinding>
    <binding  transactionFlow="false">
      <security mode="None">
        <message clientCredentialType="None" />
        <transport clientCredentialType="None"></transport>
      </security>
      <reliableSession enabled="false"/>
    </binding>
  </netTcpBinding>

We are trying to use Net.Tcp binding because of its realibility and velocity.

FACT 1 - Net.Tcp Binding is primary reason

When the load is high, the channel Net.Tcp stop working. That's it! But the BasicHttp still working like a charm.

The WindowsService: the channel net.tcp last down for some minutes (3m - 10m) before get working back (BY ITSELF, without we change anything. Goblins are working hard).

The AppFabric/IIS/WAS: the channel net.tcp keep down. Need manual restart.

The BasicHttpBinding configuration is similar to net.tcp: without any treating of the message, whitout security concerns or something like that.

FACT 2 - Without any kind of logging

We couldn't find any kind, tip, trick to figure out what's happening. I have tried Dump the memory, event logs, System.Diagnostics and nothing relevant. The most relevant tip is an Error from SMSvcHost 4.0.0.0:

An error occurred while dispatching a duplicated socket: this handle is now leaked in the process. ID: 2272 Source: System.ServiceModel.Activation.TcpWorkerProcess/62875109 Exception: System.TimeoutException: This request operation sent to http://schemas.microsoft.com/2005/12/ServiceModel/Addressing/Anonymous did not receive a reply within the configured timeout (00:01:00). The time allotted to this operation may have been a portion of a longer timeout. This may be because the service is still processing the operation or because the service was unable to send a reply message. Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client.

Server stack trace: at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Channels.ServiceChannel.SendAsyncResult.End(SendAsyncResult result) at System.ServiceModel.Channels.ServiceChannel.EndCall(String action, Object[] outs, IAsyncResult result) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeEndService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: at System.Runtime.AsyncResult.End[TAsyncResult](IAsyncResult result)
at System.ServiceModel.Activation.WorkerProcess.EndDispatchSession(IAsyncResult result) Process Name: SMSvcHost Process ID: 1532

Do you have any tip or configuration trick to help me solve this issue?

Whats the best configuration for high load scenarios?

Solution

If you generated a service reference in Visual Studio, or with the svcutil tool, make sure you always call the Close or Abort methods of your proxies. I encountered a similar problem some days ago because I forgot to call these methods.