Search code examples
node.jsbuffernode-streams

How to pipe to a Node.js stream that's a buffer, not a file?


I've been searching everywhere, and all examples lead to fs.createWriteStream which is not what I want.

I'm using the archiver package, and would like to use archive.pipe() to pipe to a writeable stream that is not a file, but rather a buffer that I can use to send to s3.putObject to write to an S3 bucket. How do I set up a buffer that I can pipe to?

When I run the code below, I get "Error: not implemented".

const stream = require('stream');
const archiver = require('archiver');

const archive = archiver('zip');
const outputStream = new stream.Writable();
outputStream.on('close', () => {
  console.log('done');
});
outputStream.on('error', err => {
  console.error(err);
});
archive.pipe(outputStream);
archive.append('Testing 1 2 3', { name: 'file1.txt' });
archive.finalize();

Solution

  • Got it!

    const bl = require('bl');
    const AWS = require('aws-sdk');
    const archiver = require('archiver');
    
    const s3 = new AWS.S3();
    const archive = archiver('zip');
    let buf;
    
    archive.pipe(
      bl((err, data) => {
        buf = data;
      })
    );
    archive.file('whoa.wav', { name: 'file1.wav' });
    archive.finalize().then(x => {
      console.log('finalized', buf);
      s3.putObject(
        {
          Bucket: 'xyz',
          Key: 'whatever.zip',
          Body: buf,
        },
        err => {
          console.log('s3 result:', err);
        }
      );
    });