Multiple URLSession dataTask on asynchronous call with completion handler causes memory goes up

Question

I'm developing an upload project in swift. I'm taking very large files (video, picture with size over 500 MB) with imagepickercontroller and dividing this file into chunks which has a size 1 MB. Then I send these chunks to remote server and make them defragment in server and I'm showing this file to user.

I have no problem if the file size is under 300 MB. But after this size, memory goes up too much and app is being crashed. Actually, in every case memory usage are raising but there is no crash.

When I watch progress on console, I see URLSession task begins. But, because of these tasks are waiting response from completion handler, the task queue is growing and memory usage goes up. Is there a way when a task begins, this task's completion handler begins too? I think if I can make task queue free concurrently, my problem solves. I'm waiting your helps.

let url:URL = URL(string: "\(addressPrefix)UploadFile")!
let session = URLSession.shared
let request = NSMutableURLRequest(url: url)
request.cachePolicy = NSURLRequest.CachePolicy.reloadIgnoringCacheData
request.httpMethod = "POST"

let bodyData = "\(metaDataID)~\(chunkIndex)~\(chunkSize)~\(chunkHash)~\(wholeTicket)~\(fileDataString)"

request.httpBody = bodyData.data(using: String.Encoding(rawValue: String.Encoding.utf8.rawValue));
request.timeoutInterval = .infinity

let task = session.dataTask(with: request as URLRequest, completionHandler: {(data, response, error) in
    guard let _:Data = data, let _:URLResponse = response, error == nil else {
       var attemptCounter = 1
       if attemptCounter <= 3 {
            completion("\(attemptCounter).attempt",chunkSize, error)
            attemptCounter += 1
        }
         return
     }
    let jsonStr = String(data: data!, encoding: String.Encoding(rawValue: String.Encoding.utf8.rawValue))
    completion(jsonStr, chunkSize, error) 
 SingletonConnectionManager.sharedConnectionDataManager.dataTasks["uploadFile"] = nil 
})  
SingletonConnectionManager.sharedConnectionDataManager.dataTasks["uploadFile"] = task 
task.resume()

---I call this URLSession task from this function in a tableview controller

 tmpConnection.uploadFile(chunk, metaDataID!, chunkIndex: chunkIndex, completion: {(result, chunkSize, error) in
   // I want to enter immediately when 'uploadFile' get called })

Answer 1

The requests aren't really waiting until all of them have been sent. When things are working correctly, each callback happens when the associated request finishes, and it wouldn't make sense for that to happen sooner, because the callback provides the response from the server (which you can't possibly get back until after the request has been fully sent out).

The problem here is that you are completely clogging up the session by starting entirely too many tasks at the same time. There's a known bug in NSURLSession that causes it to start to fall apart when you create a large number of tasks in a single session all at once. When you get too many tasks in a session, IIRC, the session stops calling callbacks entirely, and basically the session becomes unusable. (There's another Stack Overflow question in which this was discussed a couple of years ago, though I can't seem to find it right now.)

And because the tasks never complete, your app ends up leaking all the memory that you're using for the body data, which means your app just allocates more and more memory until it gets evicted.

The only way to fix this problem is to stop adding all of the requests to the session all at once. Start a few tasks at first (for at most eight parts or so), and then wait to send the next part until one of the previous parts finishes or fails. This approach will not only prevent you from bricking the NSURLSession, but also will prevent you from allocating an insane amount of memory to hold all of the request body NSData objects, which are currently all sitting in RAM at once.

I suggest keeping an NSMutableArray of NSNumber object representing each unsent chunk. That way, you know what is still left to send, and you can just loop to 8 and pull off the first 8 numbers, and send the chunks with those numbers. When a request completes successfully, grab the next number out of the array and send the chunk with that number.

Also, you shouldn't stop after a particular number of retries. Instead, when a request fails, check the failure to decide whether to retry (network failure) or give up (server error). Then use reachability to wait until a good time to try again, and try again when it says that the destination host is reachable. Cancel the upload only if the user explicitly asks you to cancel the upload by hitting a cancel button or similar. If the user asks you to cancel the upload, tear down your data structure so you don't start any new requests, then invalidate the URL session.