Waiting for all nodeJS request to be finished and add a small delay to them

Question

I have an application that is meant to interact with a remote API. I have to make around 10000 (10k) requests without having it be too fast that anti-ddos protection will block me. Means I need to add a small delay.

Code

 var promises = [];
 for(var i = 0 ; i < list.length;i++)
 {
  console.log("checking " + list[i]);
  promises.push(doRequest(token,Username));
 }

 Promise.all(promises).then(function()
 {
  console.log("done");
 },function(err){
  console.log("oops");
 });

doRequest looks like this:

function doRequest(token,username)
{
  var checkUsername = { method: 'GET',
  url: link + username,
  headers: 
   { AUTHORIZATION: 'Bearer ' + token,
     Accept: 'application/json' } };
    return new Promise(function (resolve, reject) 
    {
      request(checkUsername , function (error, res, body) {
        try
        {
          var json = JSON.parse(body);
          if(json.success)
          {
            resolve(username);
          }
          else
          {
            reject(username);
          }
        }
        catch(exception)
        {
          console.log(body);
          reject(exception);
        }
      });
    });
}

It seems that every single time I get to the JSON.parse(body), it crashes because I'm sending request way too fast for the remote server to not consider me to be a spam bot, resulting in my body simply being undefined or simply the code runs too fast resulting in the same thing.

How could I redeem this situation whilst still keeping the asynchronous nature of my code so it's still relatively fast?

Answer 1

The answer depends on how fast you want to try and make it. How many requests you are allowed to do simultaneously, etc. Here is a possible strategy. The way you want to think about this, is figure out how many concurrent requests you are allowed to make, and break the work up into synchronous blocks that all run concurrently.

// the number of simultaneous requests that wont cause problems
const allowedSimultaneous = 10

// create each chain
const chains = Array(allowedSimultaneous).fill(Promise.resolve())

// now build a synchronous list for each block
list.forEach((item, i) => {
  const chainIndex = i % allowedSimultaneous
  chains[chainIndex] = chains[chainIndex]
    .then(() => doRequest(token, Username))
})

// finally use .all() to catch when all chains are completed
Promise.all(chains).then(() => console.log('done'))

Note: This is untested code, but hopefully you get the idea. You want to build an amount of concurrent chains, each running a synchronous set of calls. This ensures that the number of concurrent requests will always be allowedSimultaneous.

No need for delays at all, as there will only ever be allowedSimultaneous calls running, but if you did need some sort of delay, you could just delay after each doRequest