Is it possible to upload to gcs using file stream buffer in nestjs?

Question

I am trying to send FormData to NestJS server to upload files to GCS.

When using Multer, I was worried about the following problems.

1. If Multer stores the temp file locally, additional server costs are incurred to store it locally.
2. When Multer stores files in memory storage, memory may overflow when many users make upload requests large files at the same time.

So I'm considering uploading to gcs using a chunked file stream buffer.

However, when I use the '@google-cloud/storage' package, there is no function related to writeSteam.

"@google-cloud/storage": "^6.9.1"

Is it not possible to upload to GCS using file stream in NodeJS environment?

Answer 1

You can use signed URLs. This will allow users to upload files directly to GCS and also supports resumable uploads.

If Multer stores the temp file locally, additional server costs are incurred to store it locally.

If you do not specify a local destination for the file, it won't be saved to the disk.

However, when I use the '@google-cloud/storage' package, there is no function related to writeSteam.

There is createWriteStream() method but the buffer will still use resources before the file is uploaded to GCS.

@Controller('gcloud')
export class GcloudController {
  constructor(private readonly gcloudService: GcloudService) {}

  @Post('upload')
  @UseInterceptors(FileInterceptor('file'))
  upload(@UploadedFile() file: Express.Multer.File) {
    return this.gcloudService.upload(file);
  } 
}

@Injectable()
export class GcloudService {
  async upload(file: Express.Multer.File) {
    const writeStream = storage
      .bucket('<>.appspot.com')
      .file('<>')
      .createWriteStream();

    writeStream.on('error', (err) => {
      console.log(err);
    });

    // use a Promise to return response when file is uploaded 
    writeStream.on('finish', () => {
      console.log('File uploaded successfully.');
    });

    writeStream.end(file.buffer);
  }
}

Using signed URLs might be a better solution in case the files are very large.