Search code examples
pythonazureazure-databricksazure-keyvaultazure-service-principal

retrieve secrets from Azure Key vault in azure databricks WITHOUT secret scope

I know, I can use secret scope in Azure Databricks and easly retrieve secrets/keys from Azure KeyVault.

But

I would like to try another way -> via a service principal.The service principal has permission to get/list secrets from my azure key vault.

My steps:

  1. I create a cluster in azure databricks and install the libraries azure-identity and azure-Keyvault

I run the following code:

from azure.keyvault.secrets import SecretClient
from azure.identity import ClientSecretCredential as cs
from azure.keyvault.keys import KeyClient
kv_URI = "https://mykeyvault.vault.azure.net/"
TENANT_ID = 'yyyyy'
CLIENT_ID = 'zzzz'
CLIENT_SECRET = 'xxxxx'
credentials = cs(    
          tenant_id=TENANT_ID,
          client_id=CLIENT_ID,    
          client_secret=CLIENT_SECRET)    
secret_client = SecretClient(vault_url=kv_URI, credential=credentials)
secretlist=secret_client.get_secret("Mysecret")

but after some minutes I get time-out error.

Do you know how can I solve the problem?

Solution

  • I registered one Azure AD application and added API permission as below:

    enter image description here

    In my key vault, I added Get/List secrets permissions for that service principal like this:

    enter image description here

    Now I created one data bricks cluster with below settings:

    enter image description here

    When I ran your code in my environment, I got secret's value successfully without any time-out error like below:

    from azure.keyvault.secrets import SecretClient
from azure.identity import ClientSecretCredential as cs
from azure.keyvault.keys import KeyClient
kv_URI = "https://mykeyvault.vault.azure.net/"
TENANT_ID = 'yyyyy'
CLIENT_ID = 'zzzz'
CLIENT_SECRET = 'xxxxx'
credentials = cs(    
          tenant_id=TENANT_ID,
          client_id=CLIENT_ID,    
          client_secret=CLIENT_SECRET)    
secret_client = SecretClient(vault_url=kv_URI, credential=credentials)
secretlist=secret_client.get_secret("Mysecret")
print(secretlist.value)

    Response:

    enter image description here

    but after some minutes I get time-out error.

    In your case, try increasing the size of your Azure Data Bricks cluster and make sure to restart it. If still the error persists, create new data bricks cluster with same settings that I mentioned above and repeat the process.