Azure Databricks - Need to replace secrets in Notebook during CI/CD process

Question

I have a requirement where I am fetching the secrets from key vault based secret scope in the Databricks Notebook.I have secrets for dev and production Azure resources e.g. dev and production instance of Data lake Gen2. Now, I have a CI/CD pipeline in place which deploys the Notebook as is, from dev to production databricks folder.

However, I have no clue of how to replace the secret names in the Notebook while deployment so that the databricks Notebook in dev point to the dev resources and secrets in production point to production resources. To accomplish this, I need to override the secret names in the CI/CD pipeline or some other design to accomplish this.

Need a solution for the same. Appreciate any leads.

Thanks in advance

Answer 1

Usual approach is to pass the secret scope name as a parameter for the notebook. To accomplish this, you need to create a text widget that will be used to pass secret scope name, and just pass dev or prod scope name as a parameter for a job. It will look something like this (in Python):

# cell 1
dbutils.widgets.text("secret_scope", "", "Secret Scope")
# cell 2
secret_scope = dbutils.widgets.get("secret_scope")
# cell 3
my_secret = dbutils.secrets.get(secret_scope, "mySecret")