Download file from Google Drive and upload to S3 using NodeJS

Question

I download my PDF file from Google drive according to the documentation:

const file = await this.driveClient.files.get(
  {
    fileId: id,
    alt: 'media',
  },
  {
    responseType: 'stream'
  },
);

Then I construct a form data:

const formData = new FormData();
formData.append('file', file.data, 'file.pdf');

And send it to S3 via the presigned upload url:

const uploadedDocument = await axios({
  method: 'put',
  url: presignedS3Url,
  data: formData,
  headers: formData.getHeaders(),
});

The flow works, but the file uploaded to s3 appears corrupted:

I also tried different response types from Google API such as blob. Any idea what I am missing? Thanks in advance!

Answer 1

I managed to solve the issue by piping the file stream from google drive directly to s3 using nodejs built-in packages.

import * as https from 'https';
import { promisify } from 'util';
import { pipeline } from 'stream';

//...

const file = await this.driveClient.files.get(
  {
    fileId: id,
    alt: 'media',
  },
  {
    responseType: 'stream'
  },
);

await promisify(pipeline)(
  file.data,
  https.request(s3SignedUrl, {
    method: 'PUT',
    headers: {
      'content-type': fileMetadata.mimeType,
      'content-length': fileMetadata.size,
    },
    timeout: 60000,
  }),
);