I will try to keep it short and precise.
Requirement:
Download large (400mb) xml response from 3rd party and store as ZipArchive on disk.
Current solution:
using (var memoryStream = new MemoryStream())
{
using (var archive = new ZipArchive(memoryStream, ZipArchiveMode.Create, true))
{
var file = archive.CreateEntry($"{deliveryDate:yyyyMMdd}.xml");
using(var entryStream = file.Open())
{
using (var payload = new MemoryStream())
{
using var response = await _httpClient.GetAsync(url, HttpCompletionOption.ResponseHeadersRead);
response.EnsureSuccessStatusCode();
await response.Content.CopyToAsync(payload);
payload.Seek(0, SeekOrigin.Begin);
await payload.CopyToAsync(entryStream);
}
}
}
using (var fileStream = new FileStream(Path.Combine(filePath), FileMode.Create, FileAccess.Write, FileShare.None))
{
memoryStream.Seek(0, SeekOrigin.Begin);
await memoryStream.CopyToAsync(fileStream);
}
}
Additional Information:
I can compress a 400mb file to approx. 20mb in about 40 seconds. 1/4 is download 3/4 is compression. The httpClient is re-used. The code runs in a long lived application hosted as a k8 linux pod.
Issues with current solution:
I fail to understand if this implementation will clean up after itself. I would be thankful for pointers towards potential leaks.
may be writing more directly to the filestream would be faster / cleaner and the response should be disposed:
using System.IO.Compression;
string url = "https://stackoverflow.com/questions/70605408/better-way-to-process-large-httpclient-response-400mb-to-ziparchive";
string filePath = "test.zip";
using(HttpClient client = new HttpClient())
using (var fileStream = new FileStream(Path.Combine(filePath), FileMode.Create, FileAccess.Write, FileShare.None))
using (var archive = new ZipArchive(fileStream, ZipArchiveMode.Create, true))
{
var file = archive.CreateEntry($"test.xml");
using (var entryStream = file.Open())
using (var response = await client.GetAsync(url, HttpCompletionOption.ResponseHeadersRead))
{
response.EnsureSuccessStatusCode();
var stream = await response.Content.ReadAsStreamAsync();
await stream.CopyToAsync(entryStream);
}
}