Search code examples
azureyamlazure-pipelines

what is the best way to move files on an azure storage account container using a yaml pipeline

Im trying to write a YAML pipeline to move a file from one folder to another on an azure storage account container. Im using the the AzureFileCopy@4 task but im not sure this is the best way. Also im not really sure how to specify things like the sourcepath so that they point to the right place as I've seen this sugggestion '$(System.DefaultWorkingDirectory)/mycontainer/source' but i dont think that is right

Solution

  • You seem to have been using a wrong task. Kindly note that the AzureFileCopy task is used more likely to upload the files from the agent path during a pipeline job rather than to copy files across containers in a storageaccount.

    Based on your requirement, there appears no such pipeline task out of the box to manage the blobs in Azure Storageaccounts as of now. However, you may consider using AzCopy to copy blobs between containers. This tool is already installed on Microsoft-hosted agents and we can also integrate the command in a AzureCLI task and set environment variables for azcopy to authenticate against the client Id($servicePrincipalId) and client secret ($servicePrincipalKey) of the service principal that is referenced by the ARM service connection.

    variables:
  ARMSvcCnn: ARMSvcCnnAuthAgainstSP&Secret
  Storageaccount: xxazstorageaccountxx
  SourceContainer: container1
  DestinationContainer: container2

pool:
  vmImage: ubuntu-latest

steps:
- task: AzureCLI@2
  inputs:
    azureSubscription: '$(ARMSvcCnn)'
    scriptType: 'bash'
    scriptLocation: 'inlineScript'
    inlineScript: |
      azcopy --version
      echo "1. Set environment variables for azcopy to authenticate against the client Id and secret of service principal that is referenced by the ARM service connection:"
      export AZCOPY_AUTO_LOGIN_TYPE=SPN
      export AZCOPY_SPA_APPLICATION_ID=$servicePrincipalId
      export AZCOPY_SPA_CLIENT_SECRET=$servicePrincipalKey
      export AZCOPY_TENANT_ID=$tenantId
      echo "2. List blobs in $(SourceContainer):"
      azcopy list https://$(Storageaccount).blob.core.windows.net/$(SourceContainer)
      echo "3. Copy files between containers:"
      azcopy copy "https://$(Storageaccount).blob.core.windows.net/$(SourceContainer)" "https://$(Storageaccount).blob.core.windows.net/$(DestinationContainer)" --recursive
    addSpnToEnvironment: true

    enter image description here

    In the sample, I want to leverage the values of $servicePrincipalId, $servicePrincipalKeyand $tenantId from the AzureCLI task. Therefore, I need to set addSpnToEnvironment property to true and make sure the ARM service connection is using client Id and secret to authenticate rather than using workload identity federation or MSI. As an option, we can of course replace the variables in the script with the service principal's actual information.

    In addition, don't forget to confirm that underlining service principal referenced by the ARM service connection in the AzureCLI task has sufficient permission to contribute to the storage account.

    By the way, Azure DevOps is simply an automated tool; there could be no build-in task based on your requirement. Still, we can test to integrate command toolset in the script, but we need to make sure the script is working locally with correct environment settings before running it in pipelines.

    Here are the documents that I have referenced. You may test with different authentication methods/commands as well. Hope the information could help.

    Authorize access to blobs with AzCopy & Microsoft Entra ID | Microsoft Learn

    Copy blobs between Azure storage accounts with AzCopy v10 | Microsoft Learn