On my application I have to download several files from Amazon S3, for that purpose I have created a function that download a single files and return and observable. I have also created a second function that I have called downloadAll. The purpose of this function is to download each S3 files sequentially.
As some of the files are big and there are a lot of files (more than 50) If I just merge all the observables I end up with a lot of timeouts from amazon because of concurrency.
What I have tried:
Merge with concurrency
let observables = self.syncItem!.files.map { (f) in
return Observable.of(f)
}
let o = Observable.from(observables).merge(maxConcurrent: 2)
Result: I get the two first files downloaded but the others are not been downloaded.
ConcatMap
return Observable.from(self.syncItem?.files).concatMap({ (file) in
return self.downloadS3File(file: file)
})
Result: Only the first file been downloaded
**DownloadS3FileFunction is ommitted ... it is working ok for a single file download and its emit a File object upon download completion **
I have searched a LOT before asking the question. Can someone please help?
If only the first file is downloaded when you use concatMap then the problem is in your downloadS3File(file:) function. It doesn't emit a completed event when it's finished downloading the file so the concatMap doesn't start the second download.
That's likely the problem with your other solution as well.