Search code examples
powershellazuredeploymentcommanddata-lake

Copy Items from one resource group to another in Azure data lake store using powershell

All I want is to copy the data from a development environment to production environment in Azure data lake store. There is not QA..

These are .CSV files the environments are nothing but different resource groups. I tried copying the data in the similar Resource Group using the command 

Move-AzureRmDataLakeStoreItem -AccountName "xyz" -path "/Product_S
ales_Data.csv" -destination "/mynewdirectory

Which worked fine, however, I want the data movement to take place between two different resource groups.

Possible solution that I have come across is by using the Export command which downloads the files in the local machine and then using the Import command and uploading them to a different resource group.

Import-AzureRmDataLakeStoreItem
Export-AzureRmDataLakeStoreItem

The reason behind using a PowerShell is to automate the process of importing the files/copying them across different environment which is nothing but automating the entire deployment process using PowerShell.

The solution mentioned above might help me in taking care of the process but I am looking for a better solution where the local machine or a VM is not required.

Solution

  • You do have a number of options, all of the below will be able to accomplish what you are looking to achieve. Keep in mind that you need to check the limitations of each and weigh the costs. For example, Azure functions have a limited time they can execute (default maximum of 5 minutes) and local storage limitations.

    1. Azure Logic Apps (drag and from config)
    2. Azure Data Factory (using the Data Lake lined service)
    3. Azure Functions (using the Data Lake REST API)
    4. You could use Azure Automation and PowerShell to automate your current approach.
    5. Use ADLCopy to copy between lakes (and other stores)

    Choosing which can be opinionated and subjective.