Search code examples
c#.netamazon-s3http-request

When the actual download occurs for a Http Request

I am downloading a zip file from Amazon S3. Then I convert this zip to MemoryStream. My code is below

AmazonS3Client s3Client = new AmazonS3Client(basicAwsCredentials, Amazon.RegionEndpoint.GetBySystemName(item.RegionEndPoint));
//This operation needs about 3-4s
response = await s3Client.GetObjectAsync(request);

var memoryStream = new MemoryStream();
//This operation needs about 8-9s
await response.ResponseStream.CopyToAsync(memoryStream);
memoryStream.Position = 0;

My connection speed about 70-80KBs. I want to know why coping to memory stream takes too much time?

My another question is when the actual download occurs in any http request? when I get the response object or when I copy the Stream from response object?

Edit

I am actually downloading a zip file from AWS S3 and using the MemoryStream to unzip the file like this-

var zipStream = new ZipInputStream(memoryStream);

Size of my zip is around 2MB. So which operation will be faster? Save the zip into file system, then extract or extract from MemoryStream directly?

Solution

  • My connection speed about 70-80KBs. I want to know why coping to memory stream takes too much time?

    Hard to answer this question. If the file you are downloading is too big, copying it to memory would incur some garbage collection which comes with the corresponding overhead of course. Obviously streaming the file to some other Stream such as a FileStream or a NetworkStream (depending on what exactly you are trying to achieve and your specific requirements) might improve the performance and even allow you to download multiple files at a time from Amazon without busting all your available RAM and forcing the Operating System into swapping the crap out of it on the disk.

    My another question is when the actual download occurs in any http request? when I get the response object or when I copy the Stream from response object?

    When you await on the CopyToAsync method call. The CopyToAsync method will copy small byte chunks from the source stream to the destination stream until it reaches the end of the source stream. But since your destination Stream is a MemoryStream you are of course losing all the benefits of this chunked copying because you end up with the entire file loaded into your process operating memory.