Search code examples
azure-devopsdatabricksazure-databricksazure-pipelines-yaml

How to create a databricks token from devops pipeline

I am trying to generate databricks token from devops pipeline. PFB, code

variables:
  azureSubscription: 
  AZURE_DATABRICKS_APP_ID: 2f0e6f879c1d
  rg: 
  databricksName: 



stages:
- stage: databricksDeployment
  jobs:
  - job: DBW
    pool:
      vmImage: 'ubuntu-latest'
    steps:
    - task: AzureCLI@2
      inputs:
        azureSubscription: $(azureSubscription)
        addSpnToEnvironment: true
        scriptType: 'pscore'
        scriptLocation: 'inlineScript'
        inlineScript: |
          #databrick cli
          # install databrick-cli
          python -m pip install --upgrade pip setuptools wheel databricks-cli
          
          $wsId=(az resource show --resource-type Microsoft.Databricks/workspaces -g $(rg) -n $(databricksName) --query id -o tsv)
          $workspaceUrl=(az resource show --resource-type Microsoft.Databricks/workspaces -g $(rg) -n $(databricksName) --query properties.workspaceUrl --output tsv)

          $workspaceUrlPost='https://'
          $workspaceUrlPost+=$workspaceUrl
          $workspaceUrlPost+='/api/2.0/token/create'
          echo "Https Url with Post: $workspaceUrlPost"

          $workspaceUrlHttps='https://'
          $workspaceUrlHttps+=$workspaceUrl
          $workspaceUrlHttps+='/'
          echo "Https Url : $workspaceUrlHttps"

          # token response for the Azure Databricks app
          $token=(az account get-access-token --resource $(AZURE_DATABRICKS_APP_ID) --query "accessToken" --output tsv)
          echo "Token retrieved: $token"

          # Get a token for the Azure management API
          $azToken=(az account get-access-token --resource https://management.core.windows.net/ --query "accessToken" --output tsv)

          # Create PAT token valid for approximately 10 minutes (600 seconds). Note the quota limit of 600 tokens.
          $pat_token_response=(curl --insecure -X POST ${workspaceUrlPost} `
            -H "Authorization: Bearer $token" `
            -H "X-Databricks-Azure-SP-Management-Token:$azToken" `
            -H "X-Databricks-Azure-Workspace-Resource-Id:$wsId" `
            -d '{\"lifetime_seconds\": 600,\"comment\": \"this is an example token\"}')
          
          
            
          # Print PAT token
          $dapiToken=($pat_token_response | ConvertFrom-Json).token_value
          echo "DATABRICKS_TOKEN: $dapiToken"

          $env:DATABRICKS_HOST=$workspaceUrlHttps
          $env:DATABRICKS_TOKEN=$dapiToken
          echo "DATABRICKS_HOST: $workspaceUrlHttps"
          echo "DATABRICKS_TOKEN: $dapiToken"
      displayName: 'Run after deployment script'

    - script: |
        echo "Starting Databricks notebook upload..."
        # Install Databricks CLI
        pip install databricks-cli
        
        # Set Databricks Host and Token
        export DATABRICKS_HOST="$DATABRICKS_HOST"
        export DATABRICKS_TOKEN="$DATABRICKS_TOKEN"

        # Authenticate with Databricks using the PAT
        echo "Authenticating with Databricks..."
        databricks configure --token <<EOF
        $DATABRICKS_HOST
        $DATABRICKS_TOKEN
        EOF

I am able to generate the DATABRICKS_HOST and I can see the URL in the log , but the $DATABRICKS_TOKEN is empty in the logs. I am unable to authenticate and getting the below error.

Databricks Host (should begin with https://): Databricks Host (should begin with https://): Databricks Host (should begin with https://): Aborted!

Is something wrong with the code. Please assist. Thanks.

Update:

variables:
  armConnection: 
  resourceID: 
  resourceGroup: 
  databricksName: 

trigger:
- development

pool:
  vmImage: 'ubuntu-latest'
  
steps:
- task: UsePythonVersion@0
  inputs:
    versionSpec: '3.x'
    addToPath: true

- checkout: self




- task: AzureCLI@2
  displayName: 'Azure CLI '
  inputs:
    azureSubscription: $(armConnection)
    scriptType: pscore
    scriptLocation: inlineScript
    inlineScript: |
     # Get Azure Databricks workspace information
     $workspaceUrl = 'https://'
     $workspaceUrl += (az resource show --resource-type Microsoft.Databricks/workspaces -g $(resourceGroup) -n $(databricksName) --query properties.workspaceUrl --output tsv)
     Write-Host "Workspace Url: $workspaceUrl"
     
     # Generate AAD Access Token for Azure Databricks service.
     $token = (az account get-access-token --resource $(resourceID) --query "accessToken" --output tsv)
     Write-Host "AAD Access Token: $token"
     
     # Generate Azure Databricks PAT.
     $adbPAT_response = (curl --request POST "$workspaceUrl/api/2.0/token/create" --header "Authorization: Bearer $token" --data '{\"lifetime_seconds\": 600, \"comment\": \"This is an example token.\"}')
     Write-Host $adbPAT_response
     $env:DATABRICKS_HOST=$workspaceUrl
     $env:DATABRICKS_TOKEN=$adbPAT_response
     echo "DATABRICKS_HOST: $workspaceUrl"
     echo "DATABRICKS_TOKEN: $adbPAT_response"

- script: |
        echo "Starting Databricks notebook upload..."
        # Install Databricks CLI
        pip install databricks-cli
        
        # Set Databricks Host and Token
        export DATABRICKS_HOST="$DATABRICKS_HOST"
        export DATABRICKS_TOKEN="$DATABRICKS_TOKEN"

        # Authenticate with Databricks using the PAT
        echo "Authenticating with Databricks..."
        databricks configure --token <<EOF
        workspaceUrl
        $DATABRICKS_TOKEN
        
        EOF
        # Specify the full paths to the source files
        common_function_path=$(Build.SourcesDirectory)/notebook.sql
        
        # Specify the full target paths in Databricks
        common_function_target_path="/Users/user/notebook.sql"
        

        # Upload notebooks to Databricks workspace
        echo "Uploading notebooks to Databricks..."
        databricks workspace import --language sql --overwrite "$common_function_path" "$common_function_target_path"
        
        echo "Notebooks uploaded successfully."
  displayName: 'Upload Databricks Notebooks'

Error

Authenticating with Databricks...
Databricks Host (should begin with https://): Token:/opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/getpass.py:91: GetPassWarning: Can not control echo on the terminal.
  passwd = fallback_getpass(prompt, stream)
Warning: Password input may be echoed.
 
Warning: Password input may be echoed.
 
Token:Token:
Warning: Password input may be echoed.
 Aborted!
Uploading notebooks to Databricks...
Error: IndexError: string index out of range
Solution

  • Follow the steps below to try to resolve the issue:

    1. When using the 'az account get-access-token' command to generate the AAD Access Token, ensure the associated user (or identity) has been added as the Azure Databricks workspace admin on the workspace Admin Settings page.

    2. In pipeline, run the following command lines on the AzureCLI@2 task.

    variables:
  armConnection: {ARM service connection name}
  resourceID: 2ff814a6-3304-4ab8-85cb-cd0e6f879c1d
  resourceGroup: {Resource group name}
  databricksName: {Databricks name}

. . .
steps:
- task: AzureCLI@2
  displayName: 'Azure CLI '
  inputs:
    azureSubscription: $(armConnection)
    scriptType: pscore
    scriptLocation: inlineScript
    inlineScript: |
     # Get Azure Databricks workspace information
     $workspaceUrl = 'https://'
     $workspaceUrl += (az resource show --resource-type Microsoft.Databricks/workspaces -g $(resourceGroup) -n $(databricksName) --query properties.workspaceUrl --output tsv)
     Write-Host "Workspace Url: $workspaceUrl"
     
     # Generate AAD Access Token for Azure Databricks service.
     $token = (az account get-access-token --resource $(resourceID) --query "accessToken" --output tsv)
     Write-Host "AAD Access Token: $token"
     
     # Generate Azure Databricks PAT.
     $adbPAT_response = (curl --request POST "$workspaceUrl/api/2.0/token/create" --header "Authorization: Bearer $token" --data '{\"lifetime_seconds\": 600, \"comment\": \"This is an example token.\"}')
     Write-Host $adbPAT_response