Best approach to upload the file via REST api from API gateway

Question

User Case: Customer can upload the file from the public REST api to our S3 bucket and then we can process the file using downstream services.

After doing some research I am able to find 3 ways to do it:

1. Uploading using OCTET-STREAM file type
2. Upload the file using form-data request
3. Upload the file using the pre-signed URL

In first 2 cases user will send the binary file and we will upload the file to S3 after file validation.

In the 3rd method user have to hit 3 apis. First API to get the S3 pre-signed URL which will give access to the user to upload the file to S3. In second hit user will upload the file to that s3 pre-signed URL. After the user complete the upload he will send the request to process the file.

Do we have any security issues with step 3? As user can misuse the pre-signed URL with malicious file.

Which of these method is best according to industry practice?

---

Details of each approach:

1. Uploading using OCTET-STREAM file type

Pros:

• This method is good to upload file types which can be opened in some application such as xlsx.
• 1 API hit. Direct file upload

Cons:

• This option is not suitable to upload multiple files. If in future we need to support multiple file upload this should be changed to multipart/form-data (A2).
• No metadata can be send as body parameter. Metadata can be send in headers.

2. Upload the file using form-data request

User will upload the file with the API request by attaching it as multipart form.

Pros

• We can send multiple files at the same time.
• We can send extra parameters in the body.

3. Upload the file using the pre-signed URL

Cons

• Customer have to hit the 3 APIs to upload the file. (2 API hits to upload and then 1 more API hit to check the process the file)

Answer 1

As per my findings industry practice often favors using pre-signed URLs. Some pointers:

Scalability: Offloading the upload process to the client reduces server load. Security: Pre-signed URLs can be configured with specific permissions and expiration times, minimizing misuse risks. Flexibility: Supports large file uploads and can handle various file types and sizes.

To mitigate security concerns with pre-signed URLs, we can ensure:

• Short expiration times for the URLs, clients check also have expired time saved.
• Strict permissions (only allowing PUT operations)
• Validation of the uploaded file after it reaches S3.