Frequent connection aborted exceptions when using CRM 2013 Organization Service

I have a problem running a WCF service that connects to the CRM: It frequently produces CommunicationObjectAbortedExceptions which leave me to wonder if I am doing something wrong. These execptions started occurring after many people started using it, on the test system it worked without problems.

But let's start at the beginning: I wrote two WCF Services that connect to the Microsoft CRM2013 Organization Service using my own library to execute queries on the CRM. These services are regularly called from the CRM which is used by roughly 100-200 people on a daily basis.

This works basically fine, but I frequently get a couple of exceptions which look like the following (see bottom of the Post for the full stacktrace):

System.ServiceModel.CommunicationObjectAbortedException: The HTTP request to 'http://crm/MyOrganization/XRMServices/2011/Organization.svc' was aborted. This may be due to the local channel being closed while the request was still in progress. If this behavior is not desired, then update your code so that it does not close the channel while request operations are still in progress.

By frequently I mean around 100 times a day, most often a couple of those exceptions are thrown every 5-30 minutes in batches of 3-6 exceptions. I have no idea why this is happening. I initialize the connection to the CRM Organization Service using the following class from my library in both services:

public class CrmManager : IDisposable
{
    private static CrmConnection s_connection;
    public static CrmConnection Connection
    {
        get
        {
            if (s_connection == null)
            {
                s_connection = new CrmConnection("CrmTvTest");
            }
            return s_connection;
        }
    }

    public static IOrganizationService ServiceProxy
    {
        get { return s_serviceProxy ?? (s_serviceProxy = new CachedOrganizationService(Connection)); }
    }

As can be seen, I connect to the Organization Service once per WCF service, using the CrmConnection to handle the connection details, which is stored in a static variable (acting as a singleton, since establishing the Connection is expensive and should not be done too often to my understanding). It is then passed to the CachedOrganizationService, which is static for the same reasons. The WCF service uses the default instance management (PerSession AFAIK), meaning there is probably 1 connection and organization service per user.

My Connection String looks like this (removed any sensible data, of course):

<connectionStrings>
    <add name="CrmTvTest" connectionString="Url=http://crm/MyOrganization; Username=user; Password=pw;"/>

I then use the connection with CrmServiceContext objects to execute queries using this method from my CrmManager class. Which is, of course, always called within a using-statement:

using (CrmServiceContext context = new CrmServiceContext(CrmManager.ServiceProxy))
{
    // do some stuff...
}

How can I prevent these exceptions from constantly occuring? I get the feeling this has to do with the Security Tokens used by the CRM connection expiring, but this shouldn't be a problem when I use the CrmConnection class. It should refresh them automatically.

Any advice would be very welcome, since I am pondering this issue for a while now.

UPDATE 1

I switched to using the Developer Extensions and using the CrmConnector class, to no avail (I updated the code above). I also tried passing the CrmConnection class directly to the CrmServiceContext:

using (CrmServiceContext context = new CrmServiceContext(CrmManager.Connection))

which led to the same problems as in this Stackoverflow Question, without using a load-balancer (we initially did, but disabled load-balancing to eliminate the possibility of it causing the problems.

Full Stacktrace:

---> System.Net.WebException: The request was aborted: The request was canceled. at System.Net.HttpWebRequest.GetResponse() at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) --- End of inner exception stack trace ---

Server stack trace: at System.ServiceModel.Channels.HttpChannelUtilities.ProcessGetResponseWebException(WebException webException, HttpWebRequest request, HttpAbortReason abortReason) at System.ServiceModel.Channels.HttpChannelFactory1.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout) at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.SecurityChannelFactory1.SecurityRequestChannel.Request(Message message, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout) at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation) at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)

Exception rethrown at [0]: at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg) at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type) at Microsoft.Xrm.Sdk.IOrganizationService.Execute(OrganizationRequest request) at Microsoft.Xrm.Sdk.Client.OrganizationServiceProxy.ExecuteCore(OrganizationRequest request) at Microsoft.Xrm.Sdk.Client.OrganizationServiceContext.Execute(OrganizationRequest request) at Microsoft.Xrm.Sdk.Linq.QueryProvider.RetrieveEntityCollection(OrganizationRequest request, NavigationSource source) at Microsoft.Xrm.Sdk.Linq.QueryProvider.Execute(QueryExpression qe, Boolean throwIfSequenceIsEmpty, Boolean throwIfSequenceNotSingle, Projection projection, NavigationSource source, List1 linkLookups, String& pagingCookie, Boolean& moreRecords) at Microsoft.Xrm.Sdk.Linq.QueryProvider.Execute[TElement](QueryExpression qe, Boolean throwIfSequenceIsEmpty, Boolean throwIfSequenceNotSingle, Projection projection, NavigationSource source, List1 linkLookups) at Microsoft.Xrm.Sdk.Linq.QueryProvider.Execute[TElement](Expression expression) at Microsoft.Xrm.Sdk.Linq.QueryProvider.System.Linq.IQueryProvider.Execute[TResult](Expression expression) at System.Linq.Queryable.SingleOrDefault[TSource](IQueryable`1 source) at CrmConnector.Entities.Contact.Get(Guid p_id, Boolean p_includeRelatedEntities) in j:\IntDev\Libraries\CrmConnector\Entities\Contact.cs:line 63 at CrmExtensionService.CrmExtension.GetPersonalizedEmailSignature(String p_contactId, String p_systemUserId) in j:\IntDev\Services\CrmExtensionService\CrmExtension.svc.cs:line 460

Solution

So, after fiddling around for about two months, we found the issue: The load balancing of the CRM FrontEnd was the culprit. I assumed this was disabled, too, with disabling the load balancing of our CRM Service, but it wasn't. Our CRM Service periodically established a connection with the Organization Service of Server 1, then got switched to Server 2 mid-operation and these exceptions occurred.

We're still trying to figure out how to get this to work with load balancing activated, but for the time being we keep it disabled to prevent these errors from popping up.

There is a similar case here on StackOverflow: Sporadic exceptions calling a web service that is load balanced. We are currently using a webHttpBinding and a quickly attempted to switch over to a basicHttpBinding but didn't get it to work (but as I said, this was just a quick attempt).