Proxy-Request: Forwarding file-upload does not upload the entire file

Question

I am currently trying to implement a proxy-endpont on my backend server.

This endpoint is supposed to forward stream a file that is being uploaded.

I've tried many things, but the following is the closest I got. The remaining problem here is that, for some reason, files are not written completely.

So, for example, uploading a 4MB file sometimes gives me a 320kb file, sometimes a 2MB file and so on. It's inconsistent. I have no idea if the http.Agent is even necessary, but without keepAlive I would face a different issue (error EPIPE).

Anyways, the following should simply pipe-forward the request:

@Post('/upload-proxy')
@Consumes('application/octet-stream')
public async uploadProxy(@Request() req: express.Request): Promise<void> {
  this.setHeader('Content-Type', 'application/octet-stream');

  const res = req.res;
  if (!res) {
    throw new Error('Response object is undefined');
  }

  const agent = new http.Agent({
    keepAlive: true,
    maxSockets: 2,
  });
  const options = {
    agent: agent,
    hostname: 'localhost',
    port: 8001,
    path: '/upload',
    method: 'POST',
    headers: {
      'Content-Type': 'application/octet-stream',
      Connection: 'keep-alive',
    },
  };

  const proxyReq = http.request(options, (proxyRes) => {
    proxyRes.pipe(res, { end: false });
  });

  req.pipe(proxyReq, { end: false });
  res.send({ status: 200 });
}

The second endpoint is supposed to simply receive the data and write it as a file but, as stated, it'll never end up complete:

@Post('/')
@Consumes('application/octet-stream')
public async upload(@Request() req: express.Request): Promise<void> {
  this.setHeader('Content-Type', 'application/octet-stream');
  const res = req.res;
  if (!res) {
    throw new Error('Response object is undefined');
  }

  const filePath = path.join('/home/sfalk/Desktop/', `uploaded-file.mp3`);

  if (fs.existsSync(filePath)) {
    fs.unlinkSync(filePath);
  }

  const append = fs.createWriteStream(filePath);
  req.pipe(append);

  req.on('end', () => {
    console.log('we are done?');
    res.send({ msg: 'done' }); // <-- This is never reached
  });
}

It doesn't make a difference to use a callback on data like:

req.on('data', (chunk) => {
  console.log('Chunk size:', chunk.length);
  append.write(chunk);
});

One additional error I am facing here is that I also do not know/understand how the connection will be terminated.

Answer 1

The solution was to wrap everything inside a Promise.

I'm not sure yet how this will behave with larger files w.r.t. timeouts but that'll probably an issue for later.

At least now files are consistently uploaded completely and everything works as "expected".

/*
 * /upload-proxy 
 */
const options = {
  hostname: 'localhost',
  port: 8001,
  path: '/upload',
  method: 'POST',
};

return new Promise(() => {
  const proxyReq = http.request(options, (proxyRes) => proxyRes.pipe(res));
  req.pipe(proxyReq);
});

/*
 * /upload 
 */

return new Promise(() => {
  const append = fs.createWriteStream(filePath);
  req.pipe(append).on('close', async () => {
    // Send back response
    res.send({ message: 'svc done' });
    // Schedule a processing task with pg-boss
    await boss.send(PROCESS_FILE, { filePath: filePath });
  });
});

Somehow I never stumbled upon an example which did it like this so I'm still not completely sure if that's how we're supposed to do it.