Force an async WebRequest to flush its buffer c#

Question

I have the following connection code:

_request = (HttpWebRequest)WebRequest.Create(complianceUrl);
_request.Method = "GET";

var authInfo = string.Format("{0}:{1}", _username, _password);
authInfo = Convert.ToBase64String(Encoding.Default.GetBytes(authInfo));
_request.Headers.Add("Authorization", "Basic " + authInfo);

// set stream parameters
_request.AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip;
_request.Headers.Add("Accept-Encoding", "gzip");
_request.Accept = "application/json";
_request.ContentType = "application/json";
_request.ReadWriteTimeout = 30000;
_request.AllowReadStreamBuffering = false;

_request.Timeout = 30; //seconds, sends 15-second heartbeat.
_asyncCallback = HandleResult;    //Setting handleResult as Callback method...
_request.BeginGetResponse(_asyncCallback, _request);    //Calling BeginGetResponse on 

This works fine, and the buffer fills with data and while I have large volumes of data this is fine. But with low volumes of data it takes a while for the buffer to fill up and I want to periodically flush the buffer if I have not had any activity in a while.

I tried to do that this way:

_request.GetRequestStream().FlushAsync();

But this is wrong as it tells me I am causing a ProtocolViolationException I guess as this is a GET Verb?

Can anyone tell me how to forcibly cause the connection to dump the buffer to the client?

Handling response code added:

private void HandleResult(IAsyncResult result)
{
    using (var response = (HttpWebResponse) _request.EndGetResponse(result))
    using (var stream = response.GetResponseStream())
    using (var memory = new MemoryStream())
    {
        var compressedBuffer = new byte[BlockSize];

        while (stream != null && stream.CanRead)
        {
            var readCount = stream.Read(compressedBuffer, 0, compressedBuffer.Length);

            // if readCount is 0, then the stream must have disconnected.  Process and abort!
            if (readCount == 0)
            {

            }
        }
    }
}

Answer 1

It is not possible to make the server side of an HTTP call send you data in a particular way. Neither HTTP nor TCP have provisions for that. You have to take what you get, or talk to the vendor.

Read does not block until a buffer is full. It gives you what arrives immediately. This behavior is well known, this is not a guess of mine.

The 3rd party service send 15 second heartbeats CRLF which will also eventually fill up the buffer.

This sounds like the service is doing what you want. If that is true, the problem must be in your code. But not in the code shown in the question. Try this:

    while (true)
    {
        var readCount = stream.Read(compressedBuffer, 0, compressedBuffer.Length);
        Console.WriteLine(readCount);
    }

This should show 2 bytes every 15 seconds. If not, the vendor is at fault. If yes, you are.

while (stream != null && stream.CanRead) this is weird because none of these conditions can become false. And even if they become false, what do you do about it?! This should be while (true) plus break on stream depletion.

// if readCount is 0, then the stream must have disconnected.

This condition means that the remote side has orderly finished their sending of the HTTP response.