Azure Databricks Release Pipeline Access Issues

Question

Situation:

I want to release Databricks Workspace, Cluster, Jobs, and Notebooks.

Everything works via VS Code and AZ Login. When I run Azure Devops Pipeline I get error below.

 Error: cannot read job: User *** does not have Admin or Manage Run or Owner or View permissions on job 419617767162993
    │ 
    │   with databricks_job.this,
    │   on **job.tf** line 1, in resource "databricks_job" "this":
    │    1: resource "databricks_job" "this" {
     

 Error: cannot read group member: /2.0/preview/scim/v2/Groups/576792616350882 is only accessible by admins.
╷
│ Error: cannot read service principal: /2.0/preview/scim/v2/ServicePrincipals/3879246038660033 is only accessible by admins.

╷
│ Error: cannot read job: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method

YAML Step:

- task: AzureCLI@2
    displayName: 'tf plan '
    inputs:
      azureSubscription: '${{ variables.backendServiceArm }}'
      scriptType: bash
      scriptLocation: inlineScript
      inlineScript: 'terraform plan -lock=false -out notebooks.state.plan'
      # scriptArguments: "-out notebooks.state.plan"
      workingDirectory: '$(System.DefaultWorkingDirectory)/terraform/'
      addSpnToEnvironment: true

job.tf

resource "databricks_job" "this" {
  # provider = databricks.etl_workspace_provider <-- tried both on/off

Trial & Error: I tried many things creating:

• new jobcluster/resusing exising clusters.
• adding provider / removinig provider
• added azure devops sp to the cluster as admin and ticked all permissions.

Simply, I want my pipeline to run without errors and I do not really want to use unity catalog or any SCIM stuff.

Answer 1

When you use a service principal as the authorization to deploy Azure Databricks Workspace, Cluster, Jobs, and Notebooks, you need to check the following permissions of the service principal:

1. At first, find the service principal of the ARM service connection (Azure Resource Manager service connection) used by the AzureCLI@2 task in your pipeline. In your Azure DevOps project, go to "Project Settings" > "Service connections", find and click on the ARM service connection to open its "Overview". Click on "Manage Service Principal" to open the associated service principal on Azure Portal. Remember the name (Display name) and ID (Application (client) ID) of the service principal.

   

2. Ensure the service principal has the Contributor role at least on the Azure Subscription.

3. To manage the resources and settings in an Azure Databricks workspace using the service principal, you also need to add the service principal into the admins group in the workspace.

   • Open the workspace Admin Settings page.

     

   • Add the Service Principal into the workspace.

     

   • Add the Service Principal to the admins group in the workspace.

     

---