How do I make a long list of http calls in serial?

Question

I'm trying to only make one http call at time but when I log the response from getUrl they are piling up and I start to get 409s (Too many requests)

function getUrl(url, i, cb) {
  const fetchUrl = `https://api.scraperapi.com?api_key=xxx&url=${url.url}`;

  fetch(fetchUrl).then(async res => {
    console.log(fetchUrl, 'fetched!');
    if (!res.ok) {
      const err = await res.text();
      throw err.message || res.statusText;
    }

    url.data = await res.text();
    cb(url);
  });
 }


let requests = urls.map((url, i) => {
  return new Promise(resolve => {
    getUrl(url, i, resolve);
  });
});

const all = await requests.reduce((promiseChain, currentTask) => {
  return promiseChain.then(chainResults =>
    currentTask.then(currentResult => [...chainResults, currentResult]),
  );
}, Promise.resolve([]));

Basically I don't want the next http to start until the previous one has finished. Otherwise I hammer their server.

BONUS POINTS: Make this work with 5 at a time in parallel.

Answer 1

Since you're using await, it would be a lot easier to use that everywhere instead of using confusing .thens with reduce. It'd also be good to avoid the explicit Promise construction antipattern. This should do what you want:

const results = [];
for (const url of urls) {
  const response = await fetch(url);
  if (!response.ok) {
    throw new Error(response); // or whatever logic you need with errors
  }
  results.push(await response.text());
}

Then your results variable will contain an array of response texts (or an error will have been thrown, and the code won't reach the bottom).

The syntax for an async function is an async keyword before the argument list, just like you're doing in your original code:

const fn = async () => {
  const results = [];
  for (const url of urls) {
    const response = await fetch(url);
    if (!response.ok) {
      throw new Error(response); // or whatever logic you need with errors
    }
    results.push(await response.text());
  }
  // do something with results
};

To have a limited number of requests at a time, make a queue system - when a request completes, recursively call a function that makes another request, something like:

const results = [];
const queueNext = async () => {
  if (!urls.length) return;
  const url = urls.shift();
  const response = await fetch(url);
  if (!response.ok) {
    throw new Error(response); // or whatever logic you need with errors
  }
  results.push(await response.text());
  await queueNext();
}
await Promise.all(Array.from({ length: 5 }, queueNext));
// do something with results