good practice to upload file to s3 using nodejs scalability and perfomance should be considered

Question

i have using multer to upload file in s3 and i have tried three methods to upload a file to s3

1. memoryStorage(default)
2. *custom S3stroage engine(by directly straming the file from request and uploading to s3)
3. Diskstroage(default)

in my case the second approach is good in perfomance wise but i think its not scalable that much.but second method saves lots of time(instead of writing to disk and then streaming from there its much more stright forward).but i dont know what will the cons of this approach.can any one suggest me a good approach.

Answer 1

You should consider using S3 Pre-signed URL

So instead of using your instance as middleman for the transaction, create an API which allow the client to retrieve a short-lived unique (s3 direct) url, which the client can use to upload the file directly to your desired location in S3 without going through your instance. I believe this is more scalable than your existing solutions.

However, there's a caveat. Your pre-signed url (as the name suggested) is "pre-authenticated", be careful on serving this information to your client, and ensure the url is shortlived enough to prevent further abuse.

Also, you might need additional implementation of listening to S3 event if you want to get notified when files are uploaded successfully to the storage. See https://docs.aws.amazon.com/lambda/latest/dg/with-s3.html