Cannot get node to pipe data to a download file correctly

Question

I'm fairly new to node and streaming, and I am having an issue when attempting to stream a large amount of data to a file on the client browser.

So for example, if on the server if i have a large file, test.txt, i can easily stream this to the client browser by setting the header attachment and piping the file to the request response as follows.

res.setHeader('Content-Type', 'text/csv');
res.setHeader('Content-disposition', 'attachment;filename=myfile.text');

fs.createReadStream('./test.txt')
.pipe(res);

When the user clicks the button, the download begins, and we see the data getting streamed to the download file. The stream takes several minutes, but during this time the client is not blocked and they can continue to do other things while the file is downloaded by the browser.

However my data is not stored in a file, I need to retrieve it one string at a time from another server. So I'm attempting to create my own read stream and push my data chunk by chunk, but it does not work, when i do something like this:

var s = new Readable();
s.pipe(res);

for(let i=0; i<=total; i++) {
  dataString = //code here to get next string needed to push
  s.push(dataString);
};
s.push(null);

With this code, when the user request the download, once the download begins, the client is blocked and cannot do any other actions until the download is completed. Also if the data takes more than 30 seconds to stream, we hit the server timeout in this case, and the download fails. With the file stream this is not an issue

How to I get this to act like a file stream and not block the client from doing other request while it downloads. Any recommendations on the best way to implement this would be appreciated.

Answer 1

I was able resolve this issue by doing something similar to here: How to call an asynchronous function inside a node.js readable stream

My basic code is as follows, and this is not blocking the client or timing out on the request as the data is continuously piped to the file download on the client side.

res.setHeader('Content-Type', 'text/csv');
res.setHeader('Content-disposition', 'attachment;filename=myfile.text');

function MyStream() {
  var rs = new Readable();
  var hitsadded = 0;

  rs._read = function() {}; // needed to avoid "Not implemented" exception

  getResults(queryString, function getMoreUntilDone(err, res) {
      if (err){
        logger.logError(err);
      }

      rs.push(res.data);
      hitsadded += res.records;

      if (res.recordsTotal > hitsadded) {
        getNextPage(query, getMoreUntilDone);
      } else {
        rs.push(null);
      }
    });
  return rs;
}

 MyStream().pipe(zlib.createGzip()).pipe(res);