Powershell -recursive in Azure Data Lake Store

Question

Do someone know how to list every file in a directory inside data lake store and sub directories? apparently the -recursive instruction does not work as it does in a normal environment

I need to run this script in Azure Data Lake Store, (which runs properly in my computer)

$Quarentine = "C:\PSTest\QUARENTINE"

$validate = "C:\PSTest\Files"

get-childitem $validate -rec -af | Where-Object {$_.FullName -notmatch "^C:\\PSTest\\Files\\(.+\\)*(XX.+)\.(.+)$"} | 
move-item -destination {"C:\PSTest\QUARENTINE\"+ $_.BaseName +("{0:yyyyMMddHHmmss}" -f (get-date)) + $_.Extension}

I am working with the command Get-AzureRmDataLakeStoreChildItem where apparently -recursive is not supported.

Can someone help me please?

Thanks

Answer 1

Here's a recursive way to do it (caveat: it doesn't scale well as it makes an API call for each sub-directory and is not parallelized, and because it stores all files into memory).

function Get-DataLakeStoreChildItemRecursive ([hashtable] $Params) {
    $AllFiles = New-Object Collections.Generic.List[Microsoft.Azure.Commands.DataLakeStore.Models.DataLakeStoreItem];
    recurseDataLakeStoreChildItem -AllFiles $AllFiles -Params $Params
    $AllFiles
}

function recurseDataLakeStoreChildItem ([System.Collections.ICollection] $AllFiles, [hashtable] $Params) {
    $ChildItems = Get-AzureRmDataLakeStoreChildItem @Params;
    $Path = $Params["Path"];
    foreach ($ChildItem in $ChildItems) {
        switch ($ChildItem.Type) {
            "FILE" {
                $AllFiles.Add($ChildItem);
            }
            "DIRECTORY" {
                $Params.Remove("Path");
                $Params.Add("Path", $Path + "/" + $ChildItem.Name);
                recurseDataLakeStoreChildItem -AllFiles $AllFiles -Params $Params;
            }
        }
    }
}

Get-DataLakeStoreChildItemRecursive -Params @{ 'Path' = '/Samples'; 'Account' = 'youradlsaccount' }