Search code examples
amazon-web-servicesamazon-s3aws-climultipart

AWS CLI not automatically using multipart upload for s3

According to AWS pages, if a file size is big enough it will automatically use multi part uploads so I can run this command

aws s3 cp large_test_file s3://DOC-EXAMPLE-BUCKET/

and the upload should be multipart.

However, I am uploading a 1gb file and it seems to be uploading it as a normal file.

Furthermore, my bucket policy has no permissions for multipart list or upload (just put object) so I don't understand how that is working.

Would like some clarity on this if it is

  • Indeed using multipart upload (without informing me)
  • How is it able to do so without me granting it permissions?
Solution

  • The documentation for Multipart Uploads mentions this near the section on permissions:

    Create Multipart Upload

    You must be allowed to perform the s3:PutObject action on an object to create multipart upload.

    The bucket owner can allow other principals to perform the s3:PutObject action.

    In other words, the PutObject permission allows PutObject, but also CreateMultipartUpload and related calls necessary to perform a multipart upload.