Search code examples
azureazure-data-factoryazure-blob-storageazcopy

How to upload a file from azure blob storage to Linux VM created on azure

I have one large file on my azure blob storage container. I want to move my file from blob storage to Linux VM created on azure> How can I do that using data factory? or any Powershell Command?

Solution

  • Please reference this blog：How to copy data to VM from blob storage, it gives you a way to solve the problem with Data Factory:

    "To anyone who might get into same problem in future, I solved my problem by using 'copy wizard' present in ADF.

    We need to install Data Management Gateway on VM and register it before we use 'copy wizard'.

    We need to specify blob storage as source and in destination we need to choose 'File Server Share' option. In 'File Server Share' option we need to specify user credentials which I suppose pipeline uses to login to VM, folder on VM where pipeline will copy the data."

    From the Azure Blog Storage document, there is another way can help you Mount Blob storage as a file system with blobfuse on Linux.

    Blobfuse is a virtual file system driver for Azure Blob storage. Blobfuse allows you to access your existing block blob data in your storage account through the Linux file system. Blobfuse uses the virtual directory scheme with the forward-slash '/' as a delimiter.

    This guide shows you how to use blobfuse, and mount a Blob storage container on Linux and access data. To learn more about blobfuse, read the details in the blobfuse repository.

    If you want to use AzCopy, you can reference this document Transfer data with AzCopy and Blob storage. You can download the AzCopy for Linux. It provided the command for upload and download files.

    For example, upload file:

    azcopy copy "<local-file-path>" "https://<storage-account-name>.<blob or dfs>.core.windows.net/<container-name>/<blob-name>"

    For PowerShell, you need to use PowerShell Core 6.x and later on all platforms. It works with Windows and Linux virtual machines using Windows PowerShell 5.1 (Windows only) or PowerShell 6 (Windows and Linux).

    You can find the PowerShell commands in this document:Quickstart: Upload, download, and list blobs by using Azure PowerShell

    Here is another link talked about Copy Files to Azure VM using PowerShell Remoting 6 (Windows and Linux).

    Hope this helps.