Search code examples
azure-data-lakeazure-databricks

Error connecting to DataLake(ADLS Gen2) store from databricks

I am trying to connect to dataLake Gen2 storage from databricks python, unfortunately I am running into error.

Code:

dbutils.fs.ls("abfss://<fsystem name>@<storage name>.dfs.core.windows.net/<folder name>")

Error Message:

Configuration property .dfs.core.windows.net not found.

I doubt if it is something to do with my mount code? Additionally I have added Tenant ID to container "manage access" using storage explorer.

here is my mount code:

configs = {"fs.azure.account.auth.type": "OAuth",
       "fs.azure.account.oauth.provider.type": "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider",
       "fs.azure.account.oauth2.client.id": "<client ID>",
       "fs.azure.account.oauth2.client.secret":  "secret",
       "fs.azure.account.oauth2.client.endpoint": "https://login.microsoftonline.com/directory id/oauth2/token"}
dbutils.fs.mount(  source = "abfss://filesystem name@<storage name>.dfs.core.windows.net/",  mount_point = /mnt/soldel",  extra_configs = configs)

Mount code ran fine, with no errors. Please suggest

Solution

  • Note: You cannot access the Azure Data Lake Gen2 account without configuring the storage account with Databricks.

    This is expected error message because you haven't configured storage account with databricks to list filesystem.

    Kindly check out the error message and see the correct process of list filesystem in Databricks. enter image description here

    For more details, refer "Databricks - Azure Data Lake Storage Gen2".

    Hope this helps.