Streamed HTTP with GZIP being buffered by StreamReader?

Question

Struggling to find anyone experiencing a similar issue or anything similar.

I'm currently consuming a stream over http (json) which has a GZip requirement, and I am experiencing a delay from when the data is sent, to when reader.ReadLine() reads it. It has been suggested to me that this could be related to the decoding keeping back data in a buffer?

This is what I have currently, it works fine apart from the delay.

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(endPoint);
request.Method = "GET";

request.PreAuthenticate = true;
request.Credentials = new NetworkCredential(username, password);

request.AutomaticDecompression = DecompressionMethods.GZip;
request.ContentType = "application/json";
request.Accept = "application/json";
request.Timeout = 30;
request.BeginGetResponse(AsyncCallback, request);

Then inside the AsyncCallback method I have:

HttpWebRequest request = result.AsyncState as HttpWebRequest;

using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result))
using (Stream stream = response.GetResponseStream())
using (StreamReader reader = new StreamReader(stream, Encoding.UTF8))
{

    while (!reader.EndOfStream)
    {
        string line = reader.ReadLine();
        if (string.IsNullOrWhiteSpace(line)) continue;

        Console.WriteLine(line);
    }
}

It just sits on reader.Readline() until more data is received, and then even holds back some of that. There are also keep-alive newlines received, these are often are read out all at once when it does decide to read something.

I have tested the stream running side by side with a curl command running, the curl command receives and decompresses the data perfectly fine.

Any insight would be terrific. Thanks,

Dan

EDIT Had no luck using the buffer size on streamreader.

new StreamReader(stream, Encoding.UTF8, true, 1)

EDIT Also had no luck updating to .NET 4.5 and using

request.AllowReadStreamBuffering = false;

Answer 1

Update: This seems to have issues over long periods of time with higher rates of volume, and should only be used on small volume where the buffer is impacting the application's functionality. I have since switched back to a StreamReader.

So this is what I ended up coming up with. This works, without the delay. This does not get buffered by automated GZip decompression.

using (HttpWebResponse response = (HttpWebResponse)request.EndGetResponse(result))
using (Stream stream = response.GetResponseStream())
using (MemoryStream memory = new MemoryStream())
using (GZipStream gzip = new GZipStream(memory, CompressionMode.Decompress))
{
    byte[] compressedBuffer = new byte[8192];
    byte[] uncompressedBuffer = new byte[8192];
    List<byte> output = new List<byte>();

    while (stream.CanRead)
    {
        int readCount = stream.Read(compressedBuffer, 0, compressedBuffer.Length);

        memory.Write(compressedBuffer.Take(readCount).ToArray(), 0, readCount);
        memory.Position = 0;

        int uncompressedLength = gzip.Read(uncompressedBuffer, 0, uncompressedBuffer.Length);

        output.AddRange(uncompressedBuffer.Take(uncompressedLength));

        if (!output.Contains(0x0A)) continue;

        byte[] bytesToDecode = output.Take(output.LastIndexOf(0x0A) + 1).ToArray();
        string outputString = Encoding.UTF8.GetString(bytesToDecode);
        output.RemoveRange(0, bytesToDecode.Length);

        string[] lines = outputString.Split(new[] { Environment.NewLine }, new StringSplitOptions());
        for (int i = 0; i < (lines.Length - 1); i++)
        {
            Console.WriteLine(lines[i]);
        }

        memory.SetLength(0);
    }
}