Moving several files from one bucket to another in AWS returns error: Timed Out

Question

I'm using s3cmd to copy over 5000 files from one bucket to a folder within another bucket.

Like this:

s3cmd mv --recursive -v s3://test.bucket/1111_stuff/ s3://actual.bucket/input/dataloader_input/

However, this keeps getting me this:

INFO: Retrieving list of remote files for s3://dataloader.bucket/1111_stuff/ ...
INFO: Summary: 5186 remote files to move
ERROR: timed out

It is stuck on Retrieving list of remote files for quite some time, and all I got out of it was an error.

Is this a problem on AWS side, or is it something I can fix? Is there any other way to do this?

Thanks.

Answer 1

This looks like the socket is timing out, you might want to try changing the value of socket_timeout to 180 which is about 3 minutes, in your .s3cfg

And if the above, does not help, you might want to read some similar questions

Can I move a file into a 'folder' inside an S3 bucket using the s3cmd mv command?

s3cmd failed too many times