We have streaming app, which essentially upload data (video chunks) on server in very tight schedule. We use http streaming to do so (via NSURLSession uploadTaskWithStreamedRequest), not websockets/direct connection. Chunk sizes are around 100Kb-300Kb. And we have an uploading problem, which is hard to trace to the roots... any advise welcome.
Problem: There is occasional spikes on upload latency for SOME files. Like 1 file out of 15-20 uploaded twice as slow as others. This happens sporadically and without any visible patters. This happens even on very good connection. And we have to get rid of that spikes when network conditions allow for stable upload - since we do video streaming even 1 of 15-20 "late" files give our users "pause" on realtime playback.
We was able to reproduce this situation in details in local network with ideal conditions. 1) Client (uploader): iPhone XR with IOS12. 2) Client uploading single file with 300Kb in size via local network via Wi-Fi. After succesfull upload client start to upload it again, so we can measure sequentional uploads statistics 3) Server: Node.js doing nothing - just receiving data and log timings. 4) Uploading code:
***
NSURLSessionConfiguration *configuration = [NSURLSessionConfiguration ephemeralSessionConfiguration];
configuration.allowsCellularAccess = YES;
configuration.HTTPShouldUsePipelining = YES;
configuration.networkServiceType = NSURLNetworkServiceTypeVoice;
***
NSInputStream* inputStream = [[NSInputStream alloc] initWithFileAtPath: pathTo300KbFile ];
[httpManager setTaskNeedNewBodyStreamBlock:^NSInputStream * _Nonnull(NSURLSession * _Nonnull session, NSURLSessionTask * _Nonnull task) {
return inputStream;
}];
***
NSMutableURLRequest *fileRequest = [NSMutableURLRequest requestWithURL:URL
cachePolicy:NSURLRequestReloadIgnoringLocalCacheData
timeoutInterval:100];
[fileRequest setHTTPMethod:@"POST"];
***
NSURLSessionUploadTask* uploadTask = [httpManager uploadTaskWithStreamedRequest:fileRequest progress:... completionHandler:...];
[uploadTask resume];
***
This code just start upload again from the beginning, after uploadTask returns success (via callback)
5) "Normal" Results of the test. This happens most of the time, you can see that average upload time of 300Kb file on local network is quite low and stable - around 0.5 seconds
2019-04-18 15:07:44.793512+0300 DVGCore_Example[2016:1384429] VidLib: #191:300kb-s1-7. Success: +0.54
2019-04-18 15:07:46.019874+0300 DVGCore_Example[2016:1384370] VidLib: #192:300kb-s1-8. Success: +0.67
2019-04-18 15:07:46.828315+0300 DVGCore_Example[2016:1383910] VidLib: #193:300kb-s1-9. Success: +0.37
2019-04-18 15:07:48.383446+0300 DVGCore_Example[2016:1383909] VidLib: #194:300kb-s1-10. Success: +0.83
2019-04-18 15:07:48.948218+0300 DVGCore_Example[2016:1384429] VidLib: #195:300kb-s1-1. Success: +0.29
2019-04-18 15:07:50.116136+0300 DVGCore_Example[2016:1383910] VidLib: #196:300kb-s1-2. Success: +0.37
2019-04-18 15:07:51.168454+0300 DVGCore_Example[2016:1384428] VidLib: #197:300kb-s1-3. Success: +0.33
2019-04-18 15:07:52.244631+0300 DVGCore_Example[2016:1384370] VidLib: #198:300kb-s1-4. Success: +0.33
Example of the server logs regarding average (with receivement timings), normal upload: https://gist.github.com/IPv6/27dc7a8e4a53b5219e39cce73691c0be
6) But for some uploads there is occasional spikes in upload time. 2-3 times longer
2019-04-18 15:16:50.492425+0300 DVGCore_Example[2022:1385864] VidLib: #124:300kb-s1-8. Success: +1.87
2019-04-18 15:22:06.395094+0300 DVGCore_Example[2027:1387221] VidLib: #200:300kb-s1-6. Success: +2.11
2019-04-18 15:23:28.329843+0300 DVGCore_Example[2027:1387610] VidLib: #281:300kb-s1-9. Success: +2.84
Example of the server logs for this upload: https://gist.github.com/IPv6/8d92b3899682c4cf697dbfb64a1b752c
To the bottom: There is no reasons for this (no network problems, etc) and seems like IOS should be able to do streaming without delays for hours. But this does not happen.
We see this in ideal local environment, and in "real world" same spikes are even more frequent.
The question is - how to get rid of them? any ideas?
Nice article, that helped to trace the reason https://forums.developer.apple.com/thread/45210
in our case wifi router used in test has problems receiving traffic