Getting error Invalid configuration value detected for fs.azure.account.key while reading file from ADLS Gen2 using service principal

I am getting error while reading file from ADLS Gen2 using service principal.

configs = {"fs.azure.account.auth.type": "OAuth",
           "fs.azure.account.oauth.provider.type": 
           "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
           "fs.azure.account.oauth2.client.id": "<your-service-client-id>",
           "fs.azure.account.oauth2.client.secret": dbutils.secrets.get(scope 
           = "<scope-name>", key = "<key-name>"),
           "fs.azure.account.oauth2.client.endpoint": 
           "https://login.microsoftonline.com/<your-directory-id>/oauth2/token"}

Below code using to read file:

df=spark.read.csv('abfss:
//tempfile@tempaccount.dfs.core.windows.net/temp.csv')

Getting below error:

Invalid configuration value detected for fs.azure.account.key

Solution

You may try the following:

Instead of the this line:

"fs.azure.account.oauth2.client.secret": dbutils.secrets.get(scope = "<scope-name>", key = "<key-name>"),

Try to use as below:

"fs.azure.account.oauth2.client.secret": "<key-name>",

Hope this helps.

If this answers your query, do click “Mark as Answer” and "Up-Vote" for the same. And, if you have any further query do let us know.