I download large amounts of data using a HttpWebRequest. As suggested on their website I pass the response stream directly to the JsonSerializer.Deserialize method.
Now I want to report progress during the download. There are enough sources on the web like this one that show how to do it.
My problem is, I don't know how to combine those two features.
My first idea was to download and report the progress, then rewind the stream and pass it to the Deserialize method, but that doesn't work, because the stream is not seekable.
My second idea (which I haven't tried yet) is to read chunks of data from the response stream to report progress and write these chunks to a MemoryStream. Then I would rewind the MemoryStream to the Deserialize method. That could work, but I would end up with the complete data in memory so the performance gain as described here is lost. So I might as well forget about the memory stream and read everything to a string.
I would be happy, if somebody could give me some hints how to solve this.
Create an implementation of Stream (let's call it ProgressStream) that wraps the HttpWebResponse stream and passes all Read requests to it, while tracking the total number of bytes that have been read. Then pass that ProgressStream instance to Deserialize.
The Json.NET Deserialize method will call Read on your ProgressStream, which will then call Read on the underlying web response stream. There will be no inefficient buffering because the bytes will be immediately passed back to Json.NET. You will be able to track how many bytes have been read so far; if you know the Content-Length of the response, you can calculate a percentage downloaded.
Here's a sample implementation: https://erictummers.com/2011/02/15/upload-progress-in-wcf/
Here's another one that's internal to ASP.NET itself: https://github.com/aspnet/AspNetWebStack/blob/master/src/System.Net.Http.Formatting/Handlers/ProgressStream.cs
And a third one: http://mel-green.com/2010/01/progressstream/