AWS SDK Java TransferManager limit download size

Question

I am currently downloading S3 folders of which name matches with prefix_ to a local TMP_FOLDER via the following code snippet:

    TransferManager transferManager = TransferManagerBuilder.standard().withS3Client(s3Instance).build();
    MultipleFileDownload multipleFileDownload = transferManager.downloadDirectory(S3_BUCKET, "prefix_", TMP_FOLDER);
    multipleFileDownload.waitForCompletion();
    transferManager.shutdownNow();

The problem is that my machine runs out of disk space because those folders I am trying to download are huge (they contain many tiny text files but in sum the size of each one of the folders is up to several GiB).

So, my question is, is there a way to limit TransferManager or MultipleFileDownload just to download up to a specific size of data?

Ideally it should download the tiny files one by one checking if with the next file it surpasses the predefined limit or not, stopping if it does (this will avoid to have any incomplete file at the end)

(Of course I am supposing that none of the text files itself will be bigger that the limit established)

Answer 1

You can use getProgress().getBytesTransferredMethod() to track the file size that is being uploaded/downloaded from S3.When you reach the file limit you can pause/abort the S3 transfer and resume later. The following links might be helpful to you:

• TransferManager
• Using transferManager examples