Chunking big class for serialization

Question

I am having a problem I've got a big class, in that class Ive 4 properties that contains other class, ant that class contain various lists and models(simple class with properties)

Class looks like:

[Serializable]
public class BackupProject 
{
    public DocumentSet Documents;
    public CompareSet MetaData;
    public RelationshipSet Relationships;
    public Dictionary<string, ItemConfig> DocumentData;
}

To serialize I'm using :

using (FileStream stream = File.Open(fullPath + "/" + backupFile, FileMode.Create))
{
     stream.Position= 0;
     var bformatter = new BinaryFormatter();
     using (ZipOutputStream zipStream = new ZipOutputStream(stream))
     {
           zipStream.SetLevel(9);
           ZipEntry zipEntry = new ZipEntry("BackupProject") {DateTime = DateTime.Now};
           zipStream.PutNextEntry(zipEntry);
           bformatter.Serialize(zipStream, documents);
     }
}

As you see I tried data compression and still I am getting:

System.Runtime.Serialization.SerializationException "The internal array cannot expand to greater than Int32.MaxValue elements."

That's why I need to chunk up this class, But have no idea how to do it witch such irregular class content.

Answer 1

If you're using C# 9 or above, you could use ChunkyMonkey Code Generator to generate a partial class for your BackupProject class, with two methods:

• IEnumerable<BackupProject> Chunk(int chunkSize) BackupProject
• MergeChunks(IEnumerable<BackupProject> chunks)

The Chunk method splits your object into multiple chunks, as long as the class has been decorated with [Chunk] attribute. Each chunk will contain a maximum of chunkSize elements within the each list, array, dictionary or collection property. Other types of property are repeated for each chunk.

The MergeChunks method merges back together several chunks into one object instance.

Your ___Set classes (in the Question) won't be chunked - but you could change them to be an array, list or collection. Alternatively, you could use the documentation to see how to implement a similar mechanism which can handle your ___Set classes.

https://www.nuget.org/packages/Gord0.ChunkyMonkey.CodeGenerator