Search code examples
c#.netmemorystreamzip

Better way to process large HttpClient response (400mb) to ZipArchive


I will try to keep it short and precise.

Requirement:

Download large (400mb) xml response from 3rd party and store as ZipArchive on disk.

Current solution:

using (var memoryStream = new MemoryStream())
{
    using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
    {
        var file = archive.CreateEntry($"{deliveryDate:yyyyMMdd}.xml");
        using(var entryStream = file.Open())
        {
            using (var payload = new MemoryStream())
            {
                using var response = await _httpClient.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
                response.EnsureSuccessStatusCode();
                await response.Content.CopyToAsync(payload);
                payload.Seek(0, SeekOrigin.Begin);

                await payload.CopyToAsync(entryStream);
            }
        }
    }

    using (var fileStream = new FileStream(Path.Combine(filePath), FileMode.Create, FileAccess.Write, FileShare.None))
    {
        memoryStream.Seek(0, SeekOrigin.Begin);
        await memoryStream.CopyToAsync(fileStream);
    }
}

Additional Information:

I can compress a 400mb file to approx. 20mb in about 40 seconds. 1/4 is download 3/4 is compression. The httpClient is re-used. The code runs in a long lived application hosted as a k8 linux pod.

Issues with current solution:

I fail to understand if this implementation will clean up after itself. I would be thankful for pointers towards potential leaks.


Solution

  • may be writing more directly to the filestream would be faster / cleaner and the response should be disposed:

    using System.IO.Compression;
    
    string url = "https://stackoverflow.com/questions/70605408/better-way-to-process-large-httpclient-response-400mb-to-ziparchive";
    string filePath = "test.zip";
    
    
    using(HttpClient client = new HttpClient())
    using (var fileStream = new FileStream(Path.Combine(filePath), FileMode.Create, FileAccess.Write, FileShare.None))
    using (var archive = new ZipArchive(fileStream, ZipArchiveMode.Create, true))
    {
        var file = archive.CreateEntry($"test.xml");
        using (var entryStream = file.Open())
        using (var response = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
        {
           response.EnsureSuccessStatusCode();
           var stream = await response.Content.ReadAsStreamAsync();
           await stream.CopyToAsync(entryStream);       
                
        }
    }