Find the content-length of the stream before uploading the file or writing the file on network

Question

I am reading the file, zipping & encrypting it and then uploading/writing on network. But I need to know the content-length of the end stream( stream returned after passing through read, zip, encryption) to make a post request.

let zlib = zlib.createGzip(),
   encrypt = crypto.cipherIV(....),
    input = fs.createReadStream('file.jpg');
function zipAndEncrypt(){
   let stream = readStream.pipe( zlib).pipe( encrypt );
   let options = {
     "stream_length":0,
     headers: { "content-type": 'image/jpeg',
                "content-length": '123456', // need to get this length 
          .....
     }
    }

// post the stream
needle( 'post', url, stream, options )
   .then( resp => { console.log( "file length", resp.body.length);})
   .catch( err => {})
}

Above code works if I enter the correct content length in headers ( in this case I knew the length ). So I need to find the length of the stream.

so far I achieved the length by :

  let chunk = [], conLength;
  stream.on( 'data', ( data ) => {
            chunk.push( data );
        } )
        .on( 'end', () => {
         conLength = Buffer.concat( chunk ).length; 
        } );

But the post request fails, SOCKET hang up error.

It looks like stream is drained or consumed as it does not emit 'data' event after finding the length using the code above.

Tried stream.resume(). But nothing works. Could you please suggest how to find the length of the stream without consuming the stream.

Answer 1

If you need to send the content length, the only way to know it, is after the file has been zipped & encrypted.

So, your solution works, but only if you send the buffer, and not the stream, because you already consumed all the data from the stream. And since you already have all the chunks in memory, you might as well send it.

let chunk = [];

stream.on('data', data => chunk.push(data))
.on('end', () => {
    const buffer = Buffer.concat(chunk);
    const conLength = buffer.length;
    // Execute the request here, sending the whole buffer, not the stream
    needle(/*...*/)
});

But if your file is too big, you're require to stream it, otherwise you will reach out of memory, an easy workaround, with a little overhead, is to pipe it to a temporary file, and then send that file. That way you can know the file size before performing the request, accessing the stream.bytesWritten property or using fs.lstat.

function zipAndEncrypt(input) {
    const gzip = zlib.createGzip();
    const encrypt = crypto.createCipheriv(algo, key, iv),

    const stream = input.pipe(gzip).pipe(encrypt);


    const fileName = tmpFileName();
    const file = fs.createWriteStream(fileName)
    stream
        .pipe(file)
        .on('finish', () => {
            let options = {
                "stream_length": 0,
                headers: {
                    "content-type": 'image/jpeg',
                    "content-length": file.bytesWritten
                }
            }

            const readStream = fs.createReadStream(fileName);

            // post the stream
            needle('post', url, readStream, options)
                .then(resp => {
                    console.log("file length", resp.body.length);
                })
                .catch(err => {})
                .finally(() => {
                    // Remove the file from disk
                });
        })

}