Search code examples
amazon-web-servicesamazon-s3file-uploadaws-sdkaws-sdk-js

Amazon retries upload with large files


I have a trouble, i'm using aws-sdk on the browser to upload videos from my web app to Amazon S3, it works fine with shortest files (less than 100MB) but with large files (500MB for example) amazon retries the upload.

For example, when the upload is in 28% it return back to 1%, i don't know why, i put an event listener of the upload file but it don't give any error, just return back to 1%

Literally this is my code:

        const params = { 
            Bucket: process.env.VUE_APP_AWS_BUCKET_NAME, // It comes from my dotenv file
            Key: videoPath, // It comes from a external var
            ACL: 'public-read',
            ContentType: this.type, // It comes from my class where i have the video type
            Body: VideoObject // <- It comes from the input file
        };

            s3.putObject(params, function (err) {
                if (err)
                    console.log(err);
            }).on('httpUploadProgress', progress => {

                console.log(progress);
                console.log(progress.loaded + " - " + progress.total);
                this.progress = parseInt((progress.loaded * 100) / progress.total);

            });

I really would like to give more info but that's all i have, i don't know why s3 retry the upload without any error (Also i don't know how to catch s3 errors...)

My internet connection is fine, i'm using my business conection and it works fine, this issue happens with all my computers


Solution

  • For large object you may try upload function that supports progress tracking and multipart uploading to upload parts in parallel. Also I didn't see content length set in your example, actually uplaod accepts a stream without a content length necessarily defined.

    An example: https://aws.amazon.com/blogs/developer/announcing-the-amazon-s3-managed-uploader-in-the-aws-sdk-for-javascript/