Search code examples
amazon-web-servicesamazon-s3aws-clitaramazon-glacier

AWS S3 Glacier upload-archive taking a long time to finish execution - ways to check status or speed upload?

I am trying to upload some .tar.gz files to AWS S3 Glacier using the upload-archive command (docs).

I ran the upload-archive command over one day ago on a 29 GB .tar.gz file like so:

aws glacier upload-archive --vault-name my-vault --account-id - --archive-description "my description" --body my-file.tar.gz

I checked today, and found it still has yet to finish execution.

My question is two parts:

  • Is there some way to see that the command is still running?
    • A progress bar would be awesome
  • Are there any ways (or alternate methods) to speed up the upload-archive?

FYI, I am using aws-cli==2.0.17. Thank you in advance for any help!

**Edit**

After running for two days, the upload-archive command errored out with the below message:

An error occurred (InvalidParameterValueException) when calling the UploadArchive operation: Invalid Content-Length: 30957118

Which lead to aws-cli #3413. The posts there are in agreement with all the answers below.

Solution

  • Take a look at multipart-upload to Glacier.

    This example initiates a multipart upload to a vault named my-vault with a part size of 1 MiB:

    aws glacier initiate-multipart-upload --account-id - --part-size 1048576 --vault-name my-vault

    As for checking if an existing upload is progressing, you can always look at the network activity on the uploading client and see if there is any bandwidth towards AWS IP addresses.

    Docs