How to efficiently set the number of bytes to download for HttpWebRequest?

Question

I'm currently working on a file downloader project. The application is designed so as to support resumable downloads. All downloaded data and its metadata(download ranges) are stored on the disk immediately per call to ReadBytes. Let's say that I used the following code snippet :-

var reader = new BinaryReader(response.GetResponseStream());
var buffr = reader.ReadBytes(_speedBuffer);
DownloadSpeed += buffr.Length;//used for reporting speed and zeroed every second

Here _speedBuffer is the number of bytes to download which is set to a default value.

I have tested the application by two methods. First is by downloading a file which is hosted on a local IIS server. The speed is great. Secondly, I tried to download the same file's copy(from where it was actually downloaded) from the internet. My net speed is real slow. Now, what I observed that if I increase the _speedBuffer then the downloading speed from the local server is good but for the internet copy, speed reporting is slow. Whereas if I decrease the value of _speedBuffer, the downloading speed(reporting) for the file's internet copy is good but not for the local server. So I thought, why shouldn't I change the _speedBuffer at runtime. But all the custom algorithms(for changing the value) I came up with were in-efficient. Means the download speed was still slow as compared other downloaders.

Is this approach OK?

Am I doing it the wrong way?

Should I stick with default value for _speedBuffer(byte count)?

Answer 1

The problem with ReadBytes in this case is that it attempts to read exactly that number of bytes, or it returns when there is no more data to read.

So you receive a packet containing 99 bytes of data, then calling ReadBytes(100) will wait for the next packet to include that missing byte.

I wouldn't use a BinaryReader at all:

byte[] buffer = new byte[bufferSize];
using (Stream responseStream = response.GetResponseStream())
{
    int bytes;
    while ((bytes = responseStream.Read(buffer, 0, buffer.Length)) > 0)
    {
        DownloadSpeed += bytes;//used for reporting speed and zeroed every second
        // on each iteration, "bytes" bytes of the buffer have been filled, store these to disk
    }

    // bytes was 0: end of stream
}