IEnumerable emit as stream?

Question

Is there a way to take an IEnumerable<T> and emit it as a readable stream kinda like this?

private void DoTheThing(IEnumerable<Foo> foos)
{
    using (var myStream = foos.EmitAsStream(f =>
    {
        var line = JsonConvert.SerializeObject(f);
        return line;
    }))
    using(var streamReader = new StreamReader(myStream))
    {
        while (!streamReader.EndOfStream)
        {
            var line = streamReader.ReadLine();
            Console.WriteLine(line);
        }
    }
}

Obviously, there are some problems with this, for example it seems to imply a StreamWriter with none specified, but the idea would be that the stream reader would just yield enumerate through the IEnumerable under the hood, apply the transform delegate, and pull the results without making any new, potentially large objects in memory.

I have some very large enumerable objects in memory, and I need to push them to external places (like Amazon S3) which accepts a stream. I really can't afford to build a MemoryStream with the collection and send that; I can spool to disk and read from disk, but I'd prefer not to if I have the option, since it seems like an extra step.

Is this possible? If it's possible, is it practical?

Answer 1

You can, achieve that by using TakeWhile, if I right understood your problem.

Something like :

while(..end ..) 
{
   var enumeration = foos.TakeWhile(..line bounds..);
   var stream = StreamFromEnum(enumeration ); //custom implementation
   // stream --> S3
}

I presume you have some custom definition of "line", which might be some sort of stride/slice of data from the stream.

A lazily-evaluated stream wrapper for IEnumerable is a one possible implementation of IEnumerable<T> -> Stream conversion.