Search code examples
node.jsasynchronousnode-streams

Node Pipes Handle Input

I am piping data from a textfile to another pipe which is downloading images from some urls. Now as expected this sends a large number of requests in quick succession and remote server shuts me down. I would like to handle next chunk only after the first is processed. My code is:

read.pipe(JSONStream.parse('*'))
.pipe(es.map(function (d, cb) {
    download_images(x,y)
       .then(function(r) ...)
       .fail(function(r)  ...)
       .fin(function(f) cb())
 })
.pipe(xyz)

Since I have just started looking into streams, I might have missed a very simple point, or in my zeal to use streams I could have ignored a better approach

  • Extremely Large json file
  • Download images with a delay
Solution

  • You can call read.pause() right before calling download_images() and then call read.resume() right before you call cb().