How to upload multiple files from directory to S3?

Question

I have a directory on an Ubuntu, with 340K images, 45GB of total size! Is there an efficient way to transfer them all to an S3 of DigitalOcean?

I thought of using s3cmd put or s3cmd sync but I'm guessing that would perform the put operation on every single file individually.

Any thoughts would be much appreciated!

Answer 1

You can use the https://github.com/mishudark/s3-parallel-put, which achieves higher throughput by multithreading but still individual files will be uploaded using single PUT operations since its happening in parallel entire operation will be done sooner. The throughput depends on factors like no of CPU, network bandwidth etc.