Search code examples
azure-databricksazure-data-factory

ADF Copy data from Azure Data Bricks Delta Lake to Azure Sql Server

I'm trying to use the data copy activity to extract information from azure databricks delta lake, but I've noticed that it doesn't pass the information directly from the delta lake to the SQL server I need, but must pass it to an azure blob storage, when running it, it throws the following error

ErrorCode=AzureDatabricksCommandError,Hit an error when running the command in Azure Databricks. Error details: Failure to initialize configurationInvalid configuration value detected for fs.azure.account.key Caused by: Invalid configuration value detected for fs.azure.account.key

Looking for information I found a possible solution but it didn't work.

Invalid configuration value detected for fs.azure.account.key copy activity fails

Does anyone have any idea how the hell to pass information from an azure databricks delta lake table to a table in Sql Server??

These are some images of the structure that I have in ADF:

enter image description here

In the image I get a message that tells me that I must have a Storage Account to continue

These are the configuration images, and execution failed:

Conf: enter image description here

Fail: enter image description here

enter image description here

Thank you very much

Solution

  • The solution for this problem was the following:

    Correct the way the Storage Access Key configuration was being defined:

    in the instruction: spark.hadoop.fs.azure.account.key..blob.core.windows.net

    The following change must be made: spark.hadoop.fs.azure.account.key. storageaccountname.dfs.core.windows.net