Out Of Memory Exception when downloading a video file

Question

We currently download video(s) from Twilio, once the video has been downloaded we save the video in AWS S3. However when deploying to production I'm currently getting:

Exception of type 'System.OutOfMemoryException' was thrown.

When trying to download the video(s),

I've checked the video(s) sizes within Twilio and they're relatively small, 172,912kb

I've upgraded the instances on S3 to large as I assumed that would be an issue as before they we're small.

however the issue still persists, the code block that it fails on is as follows:

 var request = (HttpWebRequest)WebRequest.Create($"{resource.Url}/Media");
 request.Headers.Add("Authorization", "Basic " + Convert.ToBase64String(Encoding.ASCII.GetBytes(_twilioOptions.ApiKey + ":" + _twilioOptions.ApiSecret)));
 request.AllowAutoRedirect = true;

 var responseBody = (request.GetResponseAsync().Result).GetResponseStream();

 byte[] tempBuffer = new byte[8192];
 MemoryStream ms = new MemoryStream();
 int read;
 while ((read = responseBody.Read(tempBuffer, 0, tempBuffer.Length)) > 0)
 {
    ms.Write(tempBuffer, 0, read);
 }

 var result = ms;
    
 var fileName = $"Conversations/{conversationId}/Video-{DateTime.UtcNow:yyyyMMdd-hhmmss}-{compositionSid}.{resource.Format}";
 var uploadedFile = await _fileUploader.UploadFile(result, fileName, _s3Options.SecureBucket, "video/mp4");

Can anyone recommend a solution / fix for this?

Answer 1

Theoretically you shouldn't have a problem downloading a ~170MB file into memory all at once on a S3 (7GB RAM plan). However, you are not disposing or closing any of your resources. Depending on the surrounding code this could well be the problem with unmanaged resources not being freed up and could result in out of memory over a period of time.

Now we could make explicit calls to Close() or Dispose() but its easier to just encase the relevant code in using() blocks like so:

 
 byte[] tempBuffer = new byte[8192];
 using (var responseBody = (request.GetResponseAsync().Result).GetResponseStream()) 
 using (var ms = new MemoryStream()) // <------- using block, will call Dispose() 
 {
     int read;
     while ((read = responseBody.Read(tempBuffer, 0, tempBuffer.Length)) > 0)
     {
        ms.Write(tempBuffer, 0, read);
     }

     var result = ms;
    
     var fileName = $"Conversations/{conversationId}/Video-{DateTime.UtcNow:yyyyMMdd-hhmmss}-{compositionSid}.{resource.Format}";
     var uploadedFile = await _fileUploader.UploadFile(result, fileName, 
                                                      _s3Options.SecureBucket, "video/mp4");
}

Your code is similar to this MSDN example however you will note the use of Close().

Tell me more

Thanks to mjwills for spotting the responseBody can be plonked in a using() block too, saving an explicit Close.

Going forward

Even with these fixes, you might want to utilise the best practice of streaming resources rather than loading all at once, particularly if there isn't a requirement to load it all into memory at one time. Simply create a read stream to the source and a write stream for the target. Then it is simply a matter of reading and writing blocks at a time.