Search code examples
azureazure-data-lake

How to use New-AzDataLakeGen2Item to send multiple files under a directory

Based on document from Microsoft

-Source - Specify the local source file path which will be upload to a Datalake Gen2 file.

TABLE 13
Type:   String
Position:   Named
Default value:  None
Accept pipeline input:  True
Accept wildcard characters: False  <-- no wildcard?

The doc also provided an example

PS C:\> $task = New-AzDataLakeGen2Item  -FileSystem "testfilesystem" -Path "dir1/dir2/file1" -Source "c:\sourcefile.txt" -Force -asjob

Then what if I'd like to send all files under a directory, so that

-Source "c:\temp\*" <-- wildcard not allowed
-Source "c:\temp\" <-- "Could not find file..."

instead of

-Source "c:\sourcefile.txt"
Solution

  • I don't think you will be able to upload multiple files using New-AzDataLakeGen2Item Cmdlet.

    You can however make use of azcopy tool to upload multiple files. Based on the documentation, it supports copy files from local to ADLS Gen2 account (and vice-versa) among other things.

    Please see this link for more details: https://learn.microsoft.com/en-us/azure/storage/common/storage-ref-azcopy-copy.