Search code examples
azureazure-devopsazure-databricks

Automatically copy the databricks notebook from Azure devops main repo to Workspace /live folder using Azure Devops


I am trying to copy the databricks not book from Azure devops Main Repo to Databricks Workspace Live folder. I tried the below Yaml file in my CI pipeline, the pipe line shows success but i dont see the notebook in the Workspace live folder.

trigger: none
pool:
vmImage: 'windows-latest'
  variables:
  solution: ''
  buildPlatform: 'Any CPU'
  buildConfiguration: 'Release'
steps:
  - task: CopyFiles@2
  displayName: 'Deploy Databricks notebooks'
inputs:

  SourceFolder: '$(Build.SourcesDirectory)/adb-notebook'
  TargetFolder: 'Databricks/Workspace/live'

Solution

  • DevOps task is executing on the agent machine, after CopyFiles@2 completed, the file is only copied to the target folder on agent machine, nothing changed on DevOps repo, or databricks workspace.

    To copy the files to databricks workspace/live folder, you can install extension DevOps for Azure Databricks.

    My pipeline sample:

    trigger: none
    
    pool:
      vmImage: Windows-latest
    
    steps:
    - task: configuredatabricks@0
      inputs:
        url: 'https://adb-1399576702.2.azuredatabricks.net'
        token: 'tokevalue'
    
    - task: deploynotebooks@0
      inputs:
        notebooksFolderPath: '$(System.DefaultWorkingDirectory)/adb-notebook'
        workspaceFolder: '/live'
    

    enter image description here

    PS: if you would like to copy the file to DevOps repo, you need to push the file to the repo.