How to download multiple files having same prefix as filename from various folders of S3 bucket?

Question

Let's say that I have an S3 bucket named bucketSample.

And I have different folders like abc,def and xyz.

Now I have multiple files having prefix hij_ in all the above mentioned folders.

I want to download all the files having prefix hij_. (For Example, hij_qwe.txt,hij_rty.pdf,etc)

I have gone through various ways but for GetObject I have to provide specific object names and I only know the prefix.

And using TransferManager I can download all files of folder abc but not the files with the specific prefix only.

So is there any way that I can only download all the files with prefix hij_?

Answer 1

public void getFiles(final Set<String> bucketName, final Set<String> keys, final Set<String> prefixes) {
    try {
        ObjectListing objectListing = s3Client.listObjects(bucketName); //lists all the objects in the bucket
        while (true) {
            for (Iterator<?> iterator = objectListing.getObjectSummaries().iterator();
                 iterator.hasNext(); ) {
                S3ObjectSummary summary = (S3ObjectSummary) iterator.next();
                for (String key : keys) {
                    for (String prefix : prefixes)
                        if (summary.getKey().startsWith(key + "/" prefix)) {
                            //HERE YOU CAN GET THE FULL KEY NAME AND HENCE DOWNLOAD IT IN NEW FILE USING THE TRANFER MANAGER
                        }
                    }
                }
            }
            if (objectListing.isTruncated()) {
                objectListing = s3Client.listNextBatchOfObjects(objectListing);
            } else {
                break;
            }
        }
    } catch (AmazonServiceException e) { }
}

Read about the AWS Directory Structure in here : How does AWS S3 store files? (directory structure)

Therefore, for your use case key + "/" + prefix acts as the prefix of the objects stored in the S3 bucket. By comparing the prefix will all the objects in the S3 Bucket, you can get the full key name.