Search code examples
databricksazure-databricks

Error When Mounting ADLS Gen 2 abfss from Databricks: Error IllegalArgumentException: Unsupported Azure Scheme: abfss

I have modified some code kindly provided by @Bhagyashree in an attempt to mount abfss on ADLS Gen 2

container_name = "root"
storage_account = "mystorageaccount"
key = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"

url = "abfss://" + container_name + "@" + storage_account + ".dfs.core.windows.net/"
config = "fs.azure.account.key." + storage_account + ".blob.core.windows.net"

mount_folder = "/mnt/lake"
mounted_list = dbutils.fs.mounts()

mounted_exist = False
for item in mounted_list:
  if mount_folder in item[0]:
    mounted_exist = True
    break

  if not mounted_exist:
    dbutils.fs.mount(source = url, mount_point = mount_folder, extra_configs = {config : key})

The goal is achieve a mount that looks like the following

enter image description here

But I get the error

IllegalArgumentException: Unsupported Azure Scheme: abfss

Any thoughts on why I'm getting the error?

Solution

  • IF you want to use a Service Principal (SP) for using oAuth instead:

    1. Create a Service Principal

    2. Give the SP a secret and store the secret, app-id, tenant-id

    3. Give your SP "data blob storage contributor"-role for your storageaccount

    4. better use an AzureKeyVault in Combination with Databricks ScopedCreds to use credentials!

    5. Apply this code:

    import json

container = 'my_container'
storageAccount = 'my_stoarage'
secretfile = json.loads(dbutils.secrets.get("myScope", "mySecret"))

SP_file = json.loads(dbutils.secrets.get("myScope", "my_SP"))

service_principal_id = SP_file["app-id"]
service_principal_secret = SP_file["secret"]
tenant = SP_file["tenant-id"]

configs = {
    'fs.azure.account.auth.type': 'OAuth',
    'fs.azure.account.oauth.provider.type': 'org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider',
    'fs.azure.account.oauth2.client.id': service_principal_id,
    'fs.azure.account.oauth2.client.secret': service_principal_secret,
    'fs.azure.account.oauth2.client.endpoint': f'https://login.microsoftonline.com/{tenant}/oauth2/token'
  }

dbutils.fs.mount(
    source = f'abfss://{container}@{storageAccount}.dfs.core.windows.net/',
    mount_point = '/mnt/my_storage/my_container',
    extra_configs = configs
)
```