Search code examples
pythonamazon-s3amazon-web-servicesboto

Downloading the files from s3 recursively using boto python.

I have a bucket in s3, which has deep directory structure. I wish I could download them all at once. My files look like this :

foo/bar/1. . 
foo/bar/100 . .

Are there any ways to download these files recursively from the s3 bucket using boto lib in python?

Thanks in advance.

Solution

  • You can download all files in a bucket like this (untested):

    from boto.s3.connection import S3Connection

conn = S3Connection('your-access-key','your-secret-key')
bucket = conn.get_bucket('bucket')
for key in bucket.list():
    try:
        res = key.get_contents_to_filename(key.name)
    except:
        logging.info(key.name+":"+"FAILED")

    Keep in mind that folders in S3 are simply another way of writing the key name and only clients will show this as folders.