Javascript external API calls with variable URL in for loop

Question

I am trying to call the Twitch API in a for loop where the URL being called changes with each iteration through the loop and the URL that gets called is actually one that was returned in the previous call.

Why? - The Twitch API response limits you to 100 returned streams but includes a "next" link that sets the offset so that you can get the next 100 and then the next and so on.

I am using meteor and wrote this:

var extractGames = function(limit, pages) {
    var gamesURL = 'https://api.twitch.tv/kraken/streams';
    for (i=0; i < 3; i +=1) {
        Meteor.http.get(gamesURL, {
                params: {
                    limit: limit,
                    api_version: 3
                }
            }, function (err, res) {
                if (err) {
                    console.log(res.message);
                    console.log(res.error);
                } else {
                    gamesURL = res.data._links.next;
                    var data = res.data.streams;
                    console.log(gamesURL);
                    console.log(data[0].game);
                    console.log(data[75].game);
                    console.log("-----------");
                }

            }
        )
    }
};

I believe the problem is that the new URL is not getting called each time and rather the connection to the original is staying open and not changing.

The response looks like this:

? https://api.twitch.tv/kraken/streams?limit=100&offset=100
? H1Z1
? Pokémon Red/Blue
? -----------
? https://api.twitch.tv/kraken/streams?limit=100&offset=100
? H1Z1
? Pokémon Red/Blue
? -----------
? https://api.twitch.tv/kraken/streams?limit=100&offset=100
? H1Z1
? Pokémon Red/Blue
? -----------

How do I get the loop to call the new URL each time through?

Additionally, I have tried getting the total streams and dividing by 100 (+1) and doing it differently not using the "next" URL but that code was so messy compared to this - assuming I can get this to work.

Thanks!

Answer 1

The problem here is that your requests are fired at the same time (with the same parameters!), and the variable is only changed when one returns, which is well after the first one returns. This is because the Meteor function you called is asynchronous, and does not wait. You could change it to synchronous, but I highly suggest against that because making it asynchronous is easily done.

It looks something like

A->http req

B->http req

C->http req

(wait)

A<-http return, sets var

C<-http return, sets var

C<-http return, sets var

You want to CHAIN them together, by making the next request call AFTER your first one returns, eg

function DoMeteorCall() {
    Meteor.http.get(gamesURL, {
            params: {
                limit: limit,
                api_version: 3
            }
        }, function (err, res) {
            if (err) {
                console.log(res.message);
                console.log(res.error);
            } else {
                gamesURL = res.data._links.next;
                var data = res.data.streams;
                console.log(gamesURL);
                console.log(data[0].game);
                console.log(data[75].game);
                console.log("-----------");

                // do call here
                DoMeteorCall();
            }

        }
    )
}

This will ensure that AFTER the request returns, it will call another request with the new url.