How to copy an aws private bucket to azure storage

Question

I want to copy a folder with large files in it to azure storage.

I found this article that shows how to copy a public aws bucket to azure: https://microsoft.github.io/AzureTipsAndTricks/blog/tip220.html

But how can I do this, if the aws bucket is private? How can I pass the credentials to azcopy for it to copy my files from aws bucket to azure directly?

Answer 1

From Copy data from Amazon S3 to Azure Storage by using AzCopy | Microsoft Docs:

AzCopy is a command-line utility that you can use to copy blobs or files to or from a storage account. This article helps you copy objects, directories, and buckets from Amazon Web Services (AWS) S3 to Azure blob storage by using AzCopy.

The article explains how to provide both Azure and AWS credentials.

One thing to note is that Amazon S3 cannot 'send' data to Azure Blob Storage. Therefore, something will need to call GetObject() on S3 to retrieve the data, and then send it to Azure. I'm assuming that Azure Blob Storage cannot directly request data from Amazon S3, so it means that the data will be 'downloaded' from S3, then 'uploaded' to Azure. To improve efficiency, run the AzCopy command either from an AWS or an Azure virtual machine, to reduce the latency of sending via your own computer.