Search code examples
azureazure-devopsazure-databrickscicd

Databricks notebook hardcoding

I have a databricks and a storage account each for dev and prod environment. The code promotion from the dev to prod region is done through Azure DevOps CI/CD pipelines. While in dev, the target file path is configured to dev adls which should be changed to prod adls in prod environment. Is there a way that I can remove this hardcoding and implement the code promotion using DevOps?

Solution

  • Yes, you can absolutely achieve that.

    In prod Databricks, just store them as environment variables (which can be set manually or with cluster configure

    Usually, I'll just set these variables:

    • ADLS storage name
    • Access keys

    With that, you won't need to hard-code these variables in your code. Also, it's more secure that way.

    Reference: https://docs.databricks.com/data/data-sources/azure/adls-gen2/azure-datalake-gen2-get-started.html#get-an-azure-adls-access-key