How to create a stream that receive and accumulate data while passing it to another stream in nodejs

Question

I'm trying to create a two-way stream (I believe it's called a transform stream). The stream will receive data from an http request, store the data, and pass the same data to another stream (in this case, a http.ServerResponse aka res).

Here is a code sample to demonstrate the idea:

import http from "http";

export default function handler(req, res) {
    // awkward pseudocode
    let data = [];
    let transform = new Stream();
    transform.on('data', chunk => {
        data.push(chunk);
        res.pushToStream(chunk);
    })
    transform.on('end', () => {
        res.end();
        console.log('got data: ', data)
    })
    // end awkward pseudocode
    const mapReq = http.request(`http://map.service.example.com/foo-bar`, (mapRes) => transform.pipe(res));
    mapReq.end();
}

Answer 1

Sounds like you can use PassThroughStream which is a simple implementation of a transform-stream and provides the exact functionality you're looking for. You can simply pipe the http-response to the pass-through-stream and pipe that one into your res-object.

const {PassThrough} = require('stream');

app.get("/test", function (req, res) {
    const pass = new PassThrough();
    const data = [];
    pass.on('data', (chunk) => {
        data.push(chunk)
    });
    pass.on('end', () => {
        console.log('got data: ', data)
    })
    http.request(`http://jsonplaceholder.typicode.com/users`, (response) => {
        response.pipe(pass).pipe(res);
    }).end();
});