IEnumerable, streams and WCF

Question

I am writing a service that returns large sets of data to clients. Ideally, I want to return an IEnumerable of an Entity because I want the performance benefits of the laziness, both on the Service and Client. I also want to be able to compress the stream to reduce the bandwidth.

I was able to Serialize an IEnumerable to a stream and utilize GZip to compress it. I was also able to deserialize the stream successfully. However, my implementation doesn't achieve the Laziness part of my goal.

I've read solutions to concepts similar to my question, but they all involved returning an IEnumerable of byte. Ideally, I want the client to receive an IEnumerable of Entity and be able to yield return it as it deserializes it.

[DataContract]
public class Entity
{
    [DataMember]
    public int Id { get; set; }
    [DataMember]
    public string Code { get; set; }
    [DataMember]
    public string Description { get; set; }
}

[Test]
public void TestSerialEnumGzip()
{
    var e = GetEnum();
    var s = SerializeToStreamGzip(e);
    Console.WriteLine($" TestSerialGzip stream size {s.Length}");
    var b = DeserializeFromStreamGzip<IEnumerable<Entity>>(s);
}

private IEnumerable<Entity> GetEnum()
{
    for (var x = 0; x < 10; ++x)
    {
        Console.WriteLine($"yielding {x}");
        yield return new Entity { Id = x, Code = x.ToString(), Description = x.ToString() };
    }
}

private Stream SerializeToStreamGzip<T>(T toSerialize)
{
    var s = new MemoryStream();
    using (var gz = new GZipStream(s, CompressionMode.Compress, true))
    {
        var ser = new DataContractSerializer(typeof(T));
        ser.WriteObject(gz, toSerialize);
    }
    s.Seek(0, SeekOrigin.Begin);
    return s;
}

private T DeserializeFromStreamGzip<T>(Stream stream)
{
    var ser = new DataContractSerializer(typeof(T));
    var gz = new GZipStream(stream, CompressionMode.Decompress);
    var result = (T)ser.ReadObject(gz);
    return result;
}

Answer 1

I think you might be a little confused about IEnumerable. However, that aside, you really should be focusing your research on WCF Streaming

Check out this blog Custom WCF Streaming and its associated example. It basically encapsulates everything you want and also uses BinaryFormatter,

If you wanted to take it a step further you could probably make use of Protocol Buffer Protobuf-net or add your own ad-hock compression. However, i leave those details up to you.

Basic idea is : we will have two threads, one thread will execute the complex database query and another thread will stream database rows to the clients. So we will alter the database query such that it returns only 1000 rows at time. And modify the WCF service to stream these 1000 rows to client. While WCF service is streaming database rows to the client, at the same time on a different thread, WCF Service will run the database query again to get the next 1000 rows. This way as soon as the WCF Service finishes streaming rows to the client, the next set of rows are available to stream to the client

1. WCF Client calling WCF service
2. WCF Service executing database query
3. Database returns dataset to WCF service
4. WCF Service response
5. Second database query executed by WCF service
6. WCF Stream response