I have a requirement to investigate moving files from azure data lake store folders to an azure windows virtual machine.
Just wondering what my options are - I have looked at Azcopy which looks like it might work - although I may need to shift the data into a blob store using datafactory. I'm not sure if Azcopy is able to load data from a data lake.
Got the idea from this https://learn.microsoft.com/en-us/azure/machine-learning/team-data-science-process/move-sql-server-virtual-machine
Where should my data be for BCP? While it is not required, having files containing source data located on the same machine as the target SQL Server allows for faster transfers (network speed vs local disk IO speed). You can move the flat files containing data to the machine where SQL Server is installed using various file copying tools such as AZCopy, Azure Storage Explorer or windows copy/paste via Remote Desktop Protocol (RDP).
Edit: How to copy data to Azure file service using Azure data factory - according to this, Azcopy can go to a file share via AzCopy and map the VM to there.
Is it possible to do a file copy activity using the virtual machine as the sink - if so would I need a gateway?
Edit: Looks like it is possible to copy to a vm, and a gateway is required. How to copy data to VM from blob storage?
Thanks, Tim
You could use Azure CLI 2.0 and Azure Power Shell to download files to your Windows VM.
Please see this official document.
You need install Azure CLi 2.0 on your VM firstly, you could download the msi installer file and install it directly, see this link. Using following command to download file to your local VM.
az dls fs download --account mydatalakestore --source-path /mynewfolder/vehicle1_09142014_copy.csv --destination-path "C:\mysampledata\vehicle1_09142014_copy.csv"
You also could do this with Azure Power Shell. More information about this, please refer to this link.