Send s3 file in axios request

Question

Good day, I'm getting a S3 file with aws-sdk/client s3 and I want to send it through an axios post request. I get the file, append it to a form data and send it in the request

I have written the stream to my disk for testing purposes, and the file saved was ok. Before trying with the S3 file the request was being sent with a reading directly from my disk with fs.creaReadStream and worked properly

When I sent the request with the stream in the form data axios throws me an error regarding IncomingMessage.handleStreamEnd, so my guess is that the stream is not being handled properly by the request, I have tried sending the file size in the header but is does not work

Inside my uploadFileToPlatform I see the header of multipart/form-data; boundary=---

The service platformUpload just builds and sends the axios requests

const uploadFile = async () => {

  const formData = new FormData();
    let fileTest ='testBucket/test.pdf';
    const s3Stream = await s3Service.downloadFile(fileTest);

    // formData.append('files[0].file', fs.createReadStream(pathToFile); // worked ok
    formData.append('files[0].file', s3Stream); // throws error regarding the handling the stream end
    formData.append('files[0].name', 'test.pdf');
  
    const idUpload = await platformUpload.uploadFileToPlatform(formData);
  
    return idUpload;
};

Any insight is appreciated

Answer 1

The issue you're facing might be because Axios isn't handling the stream from S3 correctly when appending it directly to the FormData. One way to fix this is to first convert the stream to a buffer and then append the buffer to the FormData. This ensures the entire file is read and ready to be sent before the request is made.

Here's how you can do it:

const { PassThrough } = require('stream');

const uploadFile = async () => {
  const formData = new FormData();
  let fileTest = 'testBucket/test.pdf';
  const s3Stream = await s3Service.downloadFile(fileTest);

  // Convert the stream to a buffer
  const buffer = await streamToBuffer(s3Stream);

  formData.append('files[0].file', buffer, { filename: 'test.pdf' });
  formData.append('files[0].name', 'test.pdf');
  
  const idUpload = await platformUpload.uploadFileToPlatform(formData);
  return idUpload;
};

const streamToBuffer = (stream) => {
  return new Promise((resolve, reject) => {
    const chunks = [];
    stream.on('data', (chunk) => chunks.push(chunk));
    stream.on('end', () => resolve(Buffer.concat(chunks)));
    stream.on('error', reject);
  });
};

In this code:

streamToBuffer function: This function reads the entire stream and concatenates it into a single buffer. uploadFile function: Instead of appending the stream directly, it appends the buffer to FormData, ensuring the file is fully ready before the POST request is made. This should resolve the issue and allow Axios to handle the file upload properly.