Why does Node-Fetch Increasingly Take Longer to Fetch in a Running Loop for 600 Reps As Loop Gets Longer?

Question

I was testing node-fetch and decided to use it in a project. In this project, I repeatedly fetch a resource, up to 600 times over the course of a couple of minutes. However, when I was testing Node-Fetch, I discovered some odd behavior: when I looped the fetch it would increasingly take longer to fetch. I proved this by generating time stamps (see them here) to the i number of iterations. This is my test code:

for(let i = 0; i < 600; i++){
    var start_time = new Date().getTime();
    fetch('http://localhost:3000').then(a=>{
        var time = (new Date().getTime() - start_time) + 'ms'
        console.log(time + ' - ' + i)
    })
}

Is there a way to work around this? I think it has something to do with threading, is there an http/https module work-around?

Also, is there a reason why the i iteration numbers are out of order?

Edit:

I realized that the reason the fetch was becoming drastically ratelimited by postman and was causing the intense wait, however, when I hosted a simple localhost app and ramped up the iterations to 60000, it took much, much longer than expected, and dropped off about 30000 iterations. Is there a reason for this? Here's the end logs:

15059ms - 15136
15059ms - 15137
15060ms - 15138
15060ms - 15140
15060ms - 15139
15061ms - 15142
15061ms - 15141
15061ms - 15143
15061ms - 15144
15062ms - 15145
15062ms - 15147
15062ms - 15146
15063ms - 15148
15063ms - 15149
15063ms - 15150
15064ms - 15152
15064ms - 15151
15064ms - 15153
15066ms - 15155
15066ms - 15154
15067ms - 15156
15067ms - 15157
15067ms - 15158
15067ms - 15159
15070ms - 15160
15073ms - 15161
15074ms - 15162
15074ms - 15163
15074ms - 15164
15074ms - 15165
15075ms - 15166
15075ms - 15167
15076ms - 15168
15076ms - 15169
15076ms - 15170
15077ms - 15171
15077ms - 15172
15077ms - 15173
15077ms - 15174
15078ms - 15175
15078ms - 15176
15078ms - 15177
15081ms - 15178
15081ms - 15179
15082ms - 15180
15082ms - 15181
15082ms - 15182
15083ms - 15183
15083ms - 15184
15083ms - 15185
15085ms - 15186
15086ms - 15187
15086ms - 15188
15086ms - 15189
15087ms - 34758

Is there a reason why: 1) the iterations were drastically under what was expected and 2) the time in ms is drastically higher than expected? Here's the code for the edited test:

for(let i = 0; i < 60000; i++){
    var start_time = new Date().getTime();
    get('http://localhost:3000').text().then(a=>{
        var time = (new Date().getTime() - start_time) + 'ms'
        console.log(time + ' - ' + i)
    })
}

Thanks!

Answer 1

As mentioned in the comments this is likely due to rate limiting on the server side. Running the same test locally will likely yield a different result.

Edit: Your localhost server will be full with requests after a while and will have to work off the load which is why it gets slower and slower over time. One rule of thumb here to help you: http requests are expensive and the first bottleneck you should avoid.

Another way to deal with this is to add a load-balancer in front for your server so you can scale up your server to multiple machines.

In the end you need to decide between a different way to get that data from the server (maybe use a socket connection that will allow you to receive only data when things have changed on the server side) or scaling your servers.