How to execute lots of network call in sequence in Swift?

Question

For example, I have an array of urls to get json data. The list count can reach to thousands. It doesn't matter the I then have a loop to retrieve json data from every element.

let urls : [URL] = getUrls();
var results = [String: Dictionary]();

for url in url {
    networkRequest (url: url, callback: { json in
        results[url.absoluteString] = json;
    });
}

The problem is that all these requests are made in concurrent time, so the results are either running very slowly and partially missing.

So I thought that was because the request ran too much at the same time. So I want to add some kind of quota of network call to run.

let urls : [URL] = getUrls();
var results = [String: Dictionary]();
var quota = 0;

for url in url {
    quota += 1;
    while quota > 5 { }
    networkRequest (url: url, callback: { json in
        quota -= 1;
        results[url.absoluteString] = json;
    });
}

Using printed console message, I can see that the first 5 network call went to the networkRequest function, but there are never any returned data. The callback was never called back. How am I supposed to implement this quota call?

Answer 1

You can try to create a serial queue

let que = DispatchQueue(label: "custQ") 
for url in urls {
  que.sync {
    quota += 1;
    while quota > 5 { }
    networkRequest (url: url, callback: { json in
        quota -= 1
        results[url.absoluteString] = json
    })
    }

}

but note networkRequest should run in same queue thread don't dispatch it , or recursively call it

var counter = 0
func callUrl(_ url:URL) {
    quota += 1;
    while quota > 5 { }
    networkRequest (url: url, callback: { json in
        self.quota -= 1
        self.results[url.absoluteString] = json
        self.counter +=1
        if self.counter < 20 {  /// or set any number 
          self.callUrl(urls[counter])
        }
    })

}