Chaining HTTP requests in Electron using request-promise

Question

UPDATE

OK so I worked it out. When using highland.js I needed a .done() to finish the stream.

var requests = [];           
_(fs.createReadStream("small.txt", { encoding: 'utf8' }))
        .splitBy('-----BEGIN-----\n')
        .splitBy('\n-----END-----\n')
        .filter(chunk => chunk !== '')
        .each(function (x) {

            requests.push(function (next) {
                Helpers.Authenticate()
                    .then(function (response1) {
                        return Helpers.Retrieve();
                    })
                    .then(function (response2) {
                        return Helpers.Retrieve();
                    })
                    .then(function () {
                        next();
                    });
            });

        })}).done(function () {
        async.waterfall(requests);
    });

The request array is now working.

---

I'm having some issues using electron and chained promises. Heres my code that being run in the main process.

var request = require('request-promise');
request.post(tppAuthenticate)
    .then(function (responseFromFirstPost) {
        var newoptions = tppRetrieveCertificate(responseFromFirstPost.APIKey)
        return request.post(newoptions) // The return is important
    })
    .then(function (responseFromSecondPost) {
        console.log(responseFromSecondPost)
    })

The entire code block executes several thousand times as its called by iterating over each line of a file. The first request gets fired continuously, but this seems to significantly block/throttle back the second request which is only getting called periodically.

I was hoping that the entire block would get called in sequence but this does not seem to be happening.

Heres my complete code block including the iteration:

        const _ = require('highland');
        const request = require('request-promise');

        fs.createReadStream(files[0], { encoding: 'utf8' }))
         .splitBy('-----BEGIN -----\n')
         .splitBy('\n-----END -----\n')
         .filter(chunk => chunk !== '')

         // .each(_.log);
         .each(function (x) {

             request.post(tppHelpers.Authenticate)
                 .then(function (responseFromFirstPost) {
                     const newoptions = tppHelpers.tppRetrieveCertificate(responseFromFirstPost.APIKey)
                     console.log(newoptions)
                     return request.post(newoptions) // The return is important
                 })
                 .then(function (responseFromSecondPost) {
                     console.log(responseFromSecondPost)
                     event.sender.send('selected-directory', responseFromSecondPost)
                 })

         });

Answer 1

If you don't want to fire every request at once, which seems to be the case reading the comments you made, then instead of running everything in parallel with:

.each(function (x) {
  // ...
});

You can instead create an empty array before you run fs.createReadStream:

var requests = [];

and in your each callback create functions to add to your array:

.each(function (x) {
  requests.push(function (next) {
    // ...
    next();
  });
});

and then you can run it in series with:

async.series(requests);

using the async module.

Just make sure that the next() is called at the right moment, e.g. in the last .then() callback of the given chain of promises.

Another way would be to use async.queue:

var queue = async.queue(function(x, callback) {
  // 
  callback();
}, 1);

(Here make sure that the callback() is called when it should. Instead of 1 at the end you could use some other number to have a certain number of requests done in parallel.)

And then in your each callback:

.each(function (x) {
  queue.push(x);
});

See the async.queue docs for more info. (Thanks to robertklep for mentioning async.queue() in the comments.)

By the way: do you even use the x in your iterations or are you just making a bunch of identical request for each line of your input?

Example

To answer your question from the comments, here is a way to construct the array of functions.

If this was your original code:

yourStream.each(function (x) {
  doRequest1()
  .then(function (response1) {
    return doRequest2();
  })
  .then(function (response2) {
    return doRequest3();
  });
});

Then you could consruct that array of functions with something like:

var requests = [];
yourStream.each(function (x) {
  requests.push(function (next) {
    doRequest1()
    .then(function (response1) {
      return doRequest2();
    })
    .then(function (response2) {
      return doRequest3();
    })
    .then(function () {   
      next();
    });
  });
});

And you could run them with:

async.series(requests);

Hope it helps.