Azure File Share synchronization

Question

I am looking for the best way to Sync files across two fileshares in Azure Storage Account. By now it looks like AZCopy is the best options, but in case of specific cases or filters it won't work.

For example:

• I want to copy files that were added after some date(last week, month etc.).
• Skip some folders/subfolders

Using Powershell is also option but it might be to slow to sync huge amount of files.

Wo you have some scripts or wrappers on top of AZCopy that can help to cover filtering by folder and file type?

Thanks Ihor

Answer 1

Here is the script that syncs some folders:

function Copy-AzureFileStorageWithPS{
<#
.SYNOPSIS
Copy a file share from a from one storage account to another using PowerShell

.DESCRIPTION
This function will copy a file share or specific folder inside of file share from one storage account to another

.PARAMETER SourceStorageAccountName

.PARAMETER DestStorageAccountName

.PARAMETER SourceResourceGroupName

.PARAMETER DestResourceGroupName

.PARAMETER FoldersToSkip

.PARAMETER FoldersToCopy

.EXAMPLE
. .\Copy-AzureFileStorageWithPS -SourceStorageAccountName  "some" -DestStorageAccountName  "other" -SourceResourceGroupName  "RG" -DestResourceGroupName  "OtherRg" 
#>

[CmdletBinding()]
param(
    [Parameter (Mandatory = $true)]
    [string] $SourceStorageAccountName,

    [Parameter (Mandatory = $true)]
    [string] $DestStorageAccountName,

    [Parameter (Mandatory = $true)]
    [string] $SourceResourceGroupName,

    [Parameter (Mandatory = $true)]
    [string] $DestResourceGroupName,

    [Parameter (Mandatory = $true)]
    [string] $FoldersToSkip = "",

    [Parameter (Mandatory = $true)]
    [string] $FoldersToCopy = ""
)

$exclusions = $FoldersToSkip.ToLower().Split(',').Trim()
$copyList = $FoldersToCopy.ToLower().Split(',').Trim()

$SourceStorageAccountKey = (Get-AzureRmStorageAccountKey -Name $SourceStorageAccountName -ResourceGroupName $SourceResourceGroupName)[0].Value
$DestStorageAccountKey = (Get-AzureRmStorageAccountKey -Name $DestStorageAccountName -ResourceGroupName $DestResourceGroupName)[0].Value

$SourceContext = New-AzureStorageContext -StorageAccountName $SourceStorageAccountName -StorageAccountKey $SourceStorageAccountKey
$DestContext = New-AzureStorageContext -StorageAccountName $DestStorageAccountName -StorageAccountKey $DestStorageAccountKey

#Get all shares from Source SA
$shares  = Get-AzureStorageshare -Context $SourceContext

foreach($share in $shares){
    $destShare = Get-AzureStorageshare -Context $DestContext -Name $share.Name -ErrorAction SilentlyContinue

    if(-Not $destShare)
    {
        $destShare = New-AzureStorageShare $share.Name -Context $DestContext            
        Write-Host "Successfully created share $($share.Name) in Account $($DestContext.StorageAccountName)"
    }

    $shareContent = Get-AzureStorageFile -Share $share     

    foreach($content in $shareContent)
    {
         if(-Not $exclusions.Contains($content.Name.ToLower())){
            if($copyList.Length -ne 0){
                if($copyList.Contains($content.Name.ToLower())){
                    Write-Output "Starting copy of $($content.Name) from $($content.Uri.AbsoluteUri) to $($destShare.Uri.AbsoluteUri+"/"+$content.Name)"
                    Copy-AonAzureFolders -SourceShare $share -SourceFolder $content -DestShare $destShare -SourceContext $SourceContext -DestContext $DestContext
                }
            }
            else
            {
                Write-Output "Starting copy of $($content.Name) from $($content.Uri.AbsoluteUri) to $($destShare.Uri.AbsoluteUri+"/"+$content.Name)"
                Copy-AonAzureFolders -SourceShare $share -SourceFolder $content -DestShare $destShare -SourceContext $SourceContext -DestContext $DestContext
            }
        }
        else{
                Write-Output "Excluding folder $($content.Name) from copying to $($destShare.Name)"
        }
    }
}
}


function Copy-AonAzureFolders
{
[CmdletBinding()]
param(
    [Parameter (Mandatory = $true)]
    [Microsoft.WindowsAzure.Storage.File.CloudFileShare] $SourceShare,

    [Parameter (Mandatory = $true)]
    [Microsoft.WindowsAzure.Storage.File.CloudFileDirectory] $SourceFolder,

    [Parameter (Mandatory = $true)]
    [Microsoft.WindowsAzure.Storage.File.CloudFileShare] $DestShare,

    [Parameter (Mandatory = $true)]
    [Microsoft.WindowsAzure.Commands.Common.Storage.AzureStorageContext] $SourceContext,

    [Parameter (Mandatory = $true)]
    [Microsoft.WindowsAzure.Commands.Common.Storage.AzureStorageContext] $DestContext
)
$pathWithoutShare = $($SourceFolder.Uri.LocalPath).Replace("/$($SourceShare.Name)","")

$shareFolders = Get-AzureStorageFile -Share $SourceShare -Path $pathWithoutShare | Get-AzureStorageFile | Where-Object {$_.GetType().Name -eq "CloudFileDirectory"}

$folderExist = Get-AzureStorageFile -Share $DestShare -Path $pathWithoutShare -ErrorAction SilentlyContinue
if(-Not $folderExist){
    $newFolder = New-AzureStorageDirectory -Share $DestShare -Path $pathWithoutShare
}

foreach($folder in $shareFolders)
{
    #Recursive call    
    Copy-AonAzureFolders -SourceShare $SourceShare -SourceFolder $folder -DestShare $DestShare -SourceContext $SourceContext -DestContext $DestContext       
}

$shareFiles = Get-AzureStorageFile -Share $SourceShare -Path $($pathWithoutShare+"/") | Get-AzureStorageFile | Where-Object {$_.GetType().Name -eq "CloudFile"}
foreach($file in $shareFiles)
{
    $fullFilePath = $pathWithoutShare + "/" + $($file.Name)

    Write-Host "-SrcShareName $($SourceShare.Name) -SrcFilePath $fullFilePath -DestShareName $($DestShare.Name) -DestFilePath $fullFilePath"
    # copy a file to the new directory
    $copyfile = Start-AzureStorageFileCopy -SrcShareName $($SourceShare.Name) -SrcFilePath $fullFilePath `
                                           -DestShareName $($DestShare.Name) -DestFilePath $fullFilePath -Context $SourceContext -DestContext $DestContext -Force

    $status = $copyfile | Get-AzureStorageFileCopyState
    while ($status.Status -eq "Pending")
    {
        $status = $copyfile | Get-AzureStorageFileCopyState
        Start-Sleep -Milliseconds 150
    }
    $copyfile
}    
}

Hope that help you but it's quite slow