Rate Limit Issue, looping through dynamic Promise requests

Question

I'm having an issue with rate limit 429 when sending to many requests to an api. I'm using the api's Node.js library to make the requests with Javascript es-6 Promises.

Each Promise takes two arguments, and the arguments change on each request.

I have solved the rate limit issue by chaining promises with .then() and including a delay function that returns a resolved promise after ??ms.

let delay = (time = delay_ms) => (result) => new Promise(resolve => setTimeout(() => resolve(result), time));

Something like this:

request(arg1, arg2) .then(delay(300)) .then(request(arg1, arg2)) .then(delay(300))...

This solved the rate limit issue BUT it's created a real headache with the amount of code I'm having to write using that solution because I'll have to write an awful amount of code.

I would like arg1 and arg2 to live in separate arrays so I can iterate through them to dynamically build the promise request and include a delay between each request.

I attempted to iterate with a forEach and for loop but the request all fire within milliseconds of each other creating the rate limit 429 issue again.

Is there a solution where:

• ALL arg1 can be stored in an array let arr1 = ['USD', 'EUR' ...]
• ALL arg2 can be stored in an array let arr2 = [60, 300, 600 ...]
• Where I can dynamically create the Promise request using arr1 & arr2 with a delay() in between each request?

my code looks something like this:

requestPromise(arg1_a, arg2_a).then(res => delay(ms)).then(requestPromise(arg1_b, arg2_b)).then(res => delay(ms))...

Any help in maybe async await? I've tried but I can't seem to get it to work with this problem? maybe due to Promises and dynamic arguments??? Not sure I'm understanding how to incorporate async await with a dynamic Promise and iteration etc...

Any help appreciated.

Answer 1

If I properly understand what you're looking for, you want to iterate through your arrays with a delay between each request, calling requestPromise(x, y) where x and y come from each of the arrays.

You can do that like this:

const delay_ms = 100;

function delay(t = delay_ms) {
    return new Promise(resolve => {
        setTimeout(resolve, t);
    });
}

function iterate(a1, a2, t) {
    let index = 0;
    const len = Math.min(a1.length, a2.length);
    if (len === 0) {
        return Promise.reject(new Error("iterate(a1, a2, t) needs two non-zero length arrays"));
    }

    function run() {
        return requestPromise(a1[index], a2[index]).then(() => {
            index++;
            // if still more to process, insert delay before next request
            if (index < len) {
                return delay(t).then(run);
            }
        });
    }

    return run();
}

// sample usage
let arr1 = ['USD', 'EUR' ...];
let arr2 = [60, 300, 600 ...];

iterate(arr1, arr2, 500).then(() => {
    // all done here
}).catch(err => {
    // got error here
});

This works by creating a promise chain where each new request is chained onto the previous promise chain and executed only after the delay. The arrays are accessed via an index variable that is initialized to 0 and then incremented after each iteration.

This function requires two non-zero length arrays and will iterate to the length of the shorter of the two arrays (if for some reason they aren't equal length).