Error : Socket hang up on multiple get requests

Question

I am developing a script on Node.js that sends a lot of requests to an API. After several requests (more than 380 requests), we receive the following error message : Error: socket hang up (code:ECONNRESET). This is a big issue for our script since we would like to send around 10000 requests. This is not an issue with the rate limit of the API because we are already handling this.

Our script is running on OVH server, and we send our requests using the package request-promise. Our version of Node.js is v 9.9.0.

Here is the function where the error is thrown :

const pollSession = async (sessionUrl) => {
    let session;
    try {
        session = await rp.get({ url: sessionUrl, json: true }, (err, res, body) => {
            if (err) {
                console.log('Err: ', err); 

            } else {
                DEBUG && console.log("Status code: ",res && res.statusCode);
                DEBUG && console.log("Status: ",res && res.body && res.body.Status);
                statusCode = res && res.statusCode;
                status = res && res.body && res.body.Status;
            }
        });
    } catch (e) {
        console.log ("----- pollSession : in catch with return value :"+e);
        return e;
    }
    return session;
}

When the request is working, we are calling this function few times in order to get the full response (because the response is huge).

When the error "Err: { Error: socket hang up" is thrown, we are calling the function again and it returns this error again. We can't afford to give up on those requests so we would like to know how to work around this error. Maybe it is possible to increase the max number of sockets (I saw it was possible with http agent, but we are using request-promise package) ?

Let me know if you need further information

Answer 1

After a lot of tests, I find out that this is related to the API I am sending requests to, Skyscanner for the record. Some flights I am searching are too long to be retrieved and lead to this error. Fixed this issue by catching the error.