Search code examples
python-3.xboto3

How to upload all files with certain extension from a local directory to AWS S3 bucket


I have a folder that contains several files, and I only want to upload the files ending with .log to a specific folder in the S3 bucket. Is there a built regex/command I can use to upload all files with specific extensions from a local directory to the AWS S3 bucket?

Or the only way to do it is to loop through the folder and upload it one by one using the s3.client of boto3?

Using wildcard as below didn't work:

s3_client.upload_file("*.log", S3_BUCKET, s3folder)

Solution

  • As @Marcin mentioned I finally needed to iterate through the file as below:

    for file in list_of_files:
        filename = os.fsdecode(file)
        if filename.endswith(".log"):
            filename = filename[2:]  # trim first 2 charcters
            logger.info('Uploading file ' + filename + ' to bucket ' + S3_BUCKET + '/' + s3folder)
            s3_client = boto3.client('s3')
            s3_client.upload_file(filename, S3_BUCKET, s3folder + '/' + filename)