copy adls gen2 blobs across subscriptions using powershell

Question

I'm trying to copy an entire folder/blob from an azure adls gen2 container in one subscription to another subscription.

Before the below code I've already connected to azure and populated hash tables of each subscription detail.

Set-AzContext -Subscription $DevDetails.SubscriptionID
$DevStorageAccount = Get-AzStorageAccount -ResourceGroupName $DevDetails.ResourceGroup -StorageAccountName $DevDetails.StorageAccount 
$DevContext = New-AzStorageContext -StorageAccountName $DevDetails.StorageAccount 
$DevStorageContainer = Get-AzStorageContainer -Name "raw" -Context $DevContext

Set-AzContext -Subscription $PPDetails.SubscriptionID
$PPStorageAccount = Get-AzStorageAccount -ResourceGroupName $PPDetails.ResourceGroup -StorageAccountName $PPDetails.StorageAccount 
$PPContext = New-AzStorageContext -StorageAccountName $PPDetails.StorageAccount 
$PPStorageContainer = Get-AzStorageContainer -Name "raw" -Context $PPContext

Set-AzContext -Subscription $ProdDetails.SubscriptionID
$PStorageAccount = Get-AzStorageAccount -ResourceGroupName $ProdDetails.ResourceGroup -StorageAccountName $ProdDetails.StorageAccount 
$PContext = New-AzStorageContext -StorageAccountName $ProdDetails.StorageAccount 
$PStorageContainer = Get-AzStorageContainer -Name "raw" -Context $PContext

I've got the variables for context for all 3 environments.

In the prod storagecontainer raw we have a gen 2 folder in a hierarchical namespace. (Folder1/Folder2/Folder3)

I would like to copy the folder and all contents from prod back to dev. Trying this command.

Start-AzStorageBlobCopy -DestContainer $DevStorageContainer.Name -Context $PContext -SrcBlob "Folder1/Folder2/Folder3/" -SrcContainer $PStorageContainer.Name -DestContext $DevContext

I get the error

Status: 403 (This request is not authorized to perform this operation.)
ErrorCode: CannotVerifyCopySource

I've also tried this.

$prodblob = Get-AzStorageBlob -Context $PContext -Container $PStorageContainer.Name -Blob "Folder1/Folder2/Folder3/"
$prodblob | Start-AzStorageBlobCopy -DestContainer $DevStorageContainer.Name -Context $DevContext

which gives the same error.

I'm able to copy it manually across using azure storage explorer and also able to do it using azcopy so it doesn't seem to be a permission issue.

Is this just not compatible with adls gen2 and I'm going about it completely wrong or is there something simple I'm missing?

Answer 1

Blob means files in this context.

I think your code can actually work if you set the -DestBlob.

EDIT: You can't copy a whole folder with the Start-AzStorageBlobCopy.

---

This is a sample code that retrieves all containers from storage account 1, and then creates the non-existing ones on storage account 2.

It then queries all files within the current container and copies them from SA1 to SA2.

Attention! the -Force option is included, so overwriting is possible. If you want to avoid this, just delete it.

#connect
Connect-AzAccount

#get all relevant information from the source
Set-AzContext -Subscription "SUB1"
$SourceResourceGroup = "SUB1_RG"
$SourceStorageAccount = "SUB1_SA"
$SourceStorageKey = (Get-AzStorageAccountKey -ResourceGroupName $SourceResourceGroup -Name $SourceStorageAccount)[0].Value
$SourceStorageContext = New-AZStorageContext -StorageAccountName $SourceStorageAccount -StorageAccountKey $SourceStorageKey

#get all relevant information from the destination
Set-AzContext -Subscription "SUB2"
$DestResourceGroup = "SUB2_RG"
$DestStorageAccount = "SUB2_SA"
$DestStorageKey = (Get-AzStorageAccountKey -ResourceGroupName $DestResourceGroup -Name $DestStorageAccount)[0].Value
$DestStorageContext = New-AZStorageContext -StorageAccountName $DestStorageAccount -StorageAccountKey $DestStorageKey

#get all container from the source
$Containers = Get-AZStorageContainer -Context $SourceStorageContext

#do something with all container
foreach ($Container in $Containers) {
    $ContainerName = $Container.Name
    #Generate new container if does not exist
    if (!((Get-AZStorageContainer -Context $DestStorageContext) | Where-Object { $_.Name -eq $ContainerName })) {   
        Write-Output "Creating new container $ContainerName"
        New-AZStorageContainer -Name $ContainerName -Permission Off -Context $DestStorageContext -ErrorAction Stop
    }

    #get all file
    $Blobs = Get-AZStorageBlob -Context $SourceStorageContext -Container $ContainerName

    #Copy all file with overwrite! just delete the "-Force" if you don't want to overwrite your existing files
    foreach ($Blob in $Blobs) {
        $BlobName = $Blob.Name
        Write-Output "Copying $BlobName from $ContainerName"
        Start-AzStorageBlobCopy -Context $SourceStorageContext -SrcContainer $ContainerName -SrcBlob $BlobName -DestContext $DestStorageContext -DestContainer $ContainerName -DestBlob $BlobName -Force -ErrorAction SilentlyContinue
    }
}

If it was helpful please upvote it or accept the answer.