Dynamics CRM Xrm Tooling error "Failed to allocate a managed memory buffer of 2147483647 bytes."

Question

I have created a XRMTool(WPF) to extract data from Dynamics CRM Online to XML file. The output file could probably have a million rows, but I am getting the following error.

Failed to allocate a managed memory buffer of 2147483647 bytes. The amount of available memory may be low.

I can add some filters and reduce the number of rows but as per the application requirement, I have to extract at least a months data in a single go, which will be around a million rows.

How can I increase the memory buffer in WPF application?

Following is my code

            string fetchaudit = @"<fetch version='1.0' mapping='logical'  output-format='xml-platform'>
        <entity name='audit'>
        <attribute name='action' />
        <attribute name='auditid' />
        <attribute name='callinguserid' />
        <attribute name='objecttypecode' />
        <attribute name='createdon' />
        <attribute name='objectid' />
        <attribute name='objectid' />
        <attribute name='operation' />
        <attribute name='regardingobjectid' />
        <attribute name='transactionid' />
        <attribute name='userid' />
        <filter type='and'>
        <condition attribute='createdon' operator='on-or-after' value='" + Fromdate + @"' />
        <condition attribute='createdon' operator='on-or-before' value='" + Todate + @"' />
        </filter>
        </entity>
       </fetch>";
     while (true)
       {
            string xml = CreateXml(fetchaudit, pagingCookie, pageNumber, fetchCount);
            RetrieveMultipleRequest fetchRequest1 = new RetrieveMultipleRequest
            {
                Query = new FetchExpression(xml) // Error occurs here when page number reaches 100
            };
            EntityCollection returnCollection = ((RetrieveMultipleResponse)_ctrl.CrmConnectionMgr.CrmSvc.Execute(fetchRequest1)).EntityCollection;

           // var collection = _ctrl.CrmConnectionMgr.CrmSvc.GetEntityDataByFetchSearchEC(xml);
            if (returnCollection != null)
           {
                if (returnCollection.Entities.Count >= 0)
                    Entities.AddRange(returnCollection.Entities);

            }
            if (returnCollection.MoreRecords)
            {
                pageNumber++;
                pagingCookie = returnCollection.PagingCookie;
            }
            else
            {
                break;
            }
        } // end while

I am using the following approach from SDK for paging SampleCode\CS\GeneralProgramming\Queries\FetchPagingWithCookie.cs (https://msdn.microsoft.com/en-us/library/gg309717.aspx)

Answer 1

Most probably the exception is not happening on the first request, if so, than reduce amount of returned rows.

Your current problem is that you are storing everything in memory till the end of retrieving

Entities.AddRange(returnCollection.Entities);

The collection of Entities is growing rapidly, but it's not failed on adding because memory allocation happening on actual SDK retrieve.

To Serialize a results of query as a single collection with DataContractSerializer you can use next IEnumerable implementation to read data by packs (and more clarifications for the code):

public class XrmEnumerator<T> : IEnumerator<T> where T : Entity
{
    private readonly Queue<T> _collected = new Queue<T>();
    private IOrganizationService _service;
    private QueryExpression _query;
    private string _lastCookie;
    private bool _moreRecords;

    public T Current { get; private set; }
    object IEnumerator.Current => Current;

    public XrmEnumerator(IOrganizationService service, QueryExpression query)
    {
        _service = service;
        _query = query;
        if (query.PageInfo == null)
            query.PageInfo = new PagingInfo
            {
                Count = 5000,
                PageNumber = 1
            };
        FillThePack();
    }

    private void FillThePack()
    {
        var result = _service.RetrieveMultiple(_query);
        _lastCookie = result.PagingCookie;
        result.Entities.ToList().ForEach(e=>_collected.Enqueue(e.ToEntity<T>()));
        _moreRecords = result.MoreRecords;
    }

    public void Dispose()
    {
        _service = null;
        _query = null;
    }

    public bool MoveNext()
    {
        if(_collected.Count == 0)
        {
            if (!_moreRecords)
            {
                return false;
            }
            _query.PageInfo.PagingCookie = _lastCookie;
            _query.PageInfo.PageNumber++;
            FillThePack();
        }
        Current = _collected.Dequeue();
        return true;
    }

    public void Reset()
    {
        _query.PageInfo.PagingCookie = string.Empty;
        _query.PageInfo.PageNumber = 1;
    }
}

public class XrmEnumerable<T> : IEnumerable<T> where T : Entity
{
    private readonly XrmEnumerator<T> _enumerator;

    public XrmEnumerable(IOrganizationService service, QueryExpression query)
    {
        _enumerator = new XrmEnumerator<T>(service, query);
    }

    public IEnumerator<T> GetEnumerator() => _enumerator;

    IEnumerator IEnumerable.GetEnumerator()
    {
        return _enumerator;
    }

    public virtual void Add(T entity)
    {
        // do your code on item deserialization
    }
}

it uses QueryExpression and not FetchXml, but you can easily convert it

usage sample:

var enumerable = new XrmEnumerable<Contact>(service, query);
var serializer = new DataContractSerializer(typeof(XrmEnumerable<Contact>));
using (var file = File.OpenWrite("contacts.xml"))
    serializer.WriteObject(file, enumerable);

Additionaly you can create a child class of XrmEnumerable and do your logic in Add function