Search code examples
azuredatabricksazure-databricks

Set spark conf for databricks cluster in python init script

Maybe is trivial but i didn't found any solutions for this, I would like to set specific azure credentials for each databricks cluster. This goes with executing the following python code in a notebook:

spark.conf.set("fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type.<storage-account>.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id.<storage-account>.dfs.core.windows.net", dbutils.secrets.get('<secrets-scope>', '<Service-principal-ID-var>'))
spark.conf.set("fs.azure.account.oauth2.client.secret.<storage-account>.dfs.core.windows.net", dbutils.secrets.get('<secrets-scope>', '<Service-principal-Secret-var>'))
spark.conf.set("fs.azure.account.oauth2.client.endpoint.<storage-account>.dfs.core.windows.net", "<endpoint>")

The basic idea for setting the credentials for each cluster is to create a cluster scoped init script (for each cluster) that execute the previous lines of code.

Is this possibile somehow?

EDIT

I've already find out and tested that this conf could be set directly in the cluster advanced options. Anyway I would prefer not to left any secret hardcoded here.

Here an example for reference:

spark conf in advanced options

Solution

  • I would prefer not to left any secret hardcoded here.

    You can use Databricks secret scope in the Spark config by specifying them in {{}}.

    For whatever secrets you don't want to hard code, create Azure Key vault secret and Databrick secret scope for all.

    My sample:

    enter image description here

    Secret scope:

    enter image description here

    For sample I have used the below secret.

    enter image description here

    As per the Documentation, specify the secret like {{secrets/<secret-scope-name>/<secret-name>}}. Follow the below code example from the same Documentation.

    fs.azure.account.auth.type.<storage-account>.dfs.core.windows.net OAuth
fs.azure.account.oauth.provider.type.<storage-account>.dfs.core.windows.net org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
fs.azure.account.oauth2.client.id.<storage-account>.dfs.core.windows.net <application-id>
fs.azure.account.oauth2.client.secret.<storage-account>.dfs.core.windows.net {{secrets/<secret-scope-name>/<secret-name>}}
fs.azure.account.oauth2.client.endpoint.<storage-account>.dfs.core.windows.net https://login.microsoftonline.com/<directory-id>/oauth2/token