How to read ChunkedEncodingReadStream in .net core

Question

I am downloading a file using HttpClient and trying to process it in chunks of 4MB. The client returns the stream as ChunkedEncodingReadStream. The file I am trying to download is 50MB in size. However, when I try to read it in a 4MB buffer using ReadAsync, I noticed it returns read count of only 1321 bytes.

My question is: how do I read the stream properly?

My code:

var client = _httpClientFactory.CreateClient();
var response = await client.GetAsync(fileUrl, HttpCompletionOption.ResponseHeadersRead);
stream = await response.Content.ReadAsStreamAsync().ConfigureAwait(false);

var chunkSize = 4096 * 1024;
var buffer = new byte[chunkSize];

var byteRead = stream.Read(buffer, 0, chunkSize);

if (byteRead < chunkSize)
{
    // process the buffer as the only chunk
}
else
{
    // process initial chunk

    while (byteRead < chunkSize)
    {
        byteRead = stream.Read(buffer, 0, chunkSize);
        // process next chunks
    }
}

Obviously, this code assumes the response stream will continue to give 4MB chunks until the last chunk is reached. But the stream I received doesn't work like that. In this scenario, what will be the correct way to process this stream?

Answer 1

I realized that the stream can return any number of bytes and only when it returns 0 bytes the stream can be considered finished as mentioned in the comment. But since my use case requires reading the stream in a specified chunk size I came up with the following code to read it -

public static async Task<(int readCount, byte[] buffer)> ReadChunkAsync(Stream stream, int chunkSize)
{
    var buffer = new byte[chunkSize];
    var readCount = 0;

    while (readCount < chunkSize)
    {
        var bytesRead = await stream.ReadAsync(buffer, readCount, chunkSize - readCount);

        if (bytesRead == 0)
        {
            break;
        }

        readCount += bytesRead;
    }

    return (readCount, buffer);
}