System.OutOfMemory exception when calling MemoryStream.ToArray() after serializing object with DataContractSerializer

Question

I am getting an intermittent "out of memory" exception at this statement:

        return ms.ToArray();

In this method:

public static byte[] Serialize(Object inst)
{
    Type t = inst.GetType();
    DataContractSerializer dcs = new DataContractSerializer(t);
    MemoryStream ms = new MemoryStream();
    dcs.WriteObject(ms, inst);
    return ms.ToArray();
}

How can I prevent it? Is there a better way to do this?

The length of ms is 182,870,206 bytes (174.4 MB)

I am putting this into a byte array so that I can then run it through compression and store it to disk. The data is (obviously) a large list of a custom class that I am downloading from a WCF server when my silverlight application starts. I am serializing it and compressing it so it uses only about 6MB in isolated storage. The next time the user visits and runs the silverlight application from the web, I check the timestamp, and if good I just open the file from isolated, decompress it, deserialize it, and load my structure. I am keeping the entire structure in memory because the application is mostly geared around manipulating the contents of this structure.

@configurator is correct. The size of the array was too big. I rolled by own serializer, by declaring a byte array of [list record count * byte count per record], then stuffed it directly myself using statements like this to stuff it:

    Buffer.BlockCopy(
           BitConverter.GetBytes(node.myInt),0,destinationArray,offset,sizeof(int)); 
    offset += sizeof(int);

and this to get it back:

    newNode.myInt= BitConverter.ToInt32(sourceByteArray,offset); 
    offset += sizeof(int);

Then I compressed it and stored it to isolated storage.

My size went from 174MB with the DataContractSerializer to 14MB with mine. After compression it went from a 6MB to a 1MB file in isolated storage.

Thanks to Configurator and Filip for their help.

Answer 1

The problem seems to be that you're expecting to return a 180MB byte array. That means the framework would need to find and allocate a consecutive 180MB of free memory to copy the stream data into, which is usually quite hard - hence the OutOfMemoryException. If you need to continue handling this amount of memory, use the memory stream itself (reading and writing to it as you need) to hold the buffer; otherwise, save it to a file (or to whatever other place you need it, e.g. serving it over a network) directly instead of using the memory stream.

I should mention that the memory stream has a 180MB array of its own in there as well, so is also in a bit of trouble and could cause OutOfMemory during serialization - it would likely be better (as in, more robust) if you could serialize it to a temporary file. You might also want to consider a more compact - but possibly less readable - serialization format, like json, binary serialization, or protocol buffers.

---

In response to the comment: to serialize directly to disk, use a FileStream instead of a MemoryStream:

public static void Serialize(Object inst, string filename)
{
    Type t = inst.GetType();
    DataContractSerializer dcs = new DataContractSerializer(t);
    using (FileStream stream = File.OpenWrite(filename)) {
        dcs.WriteObject(ms, inst);
    }
}