Writing a polars dataframe to a delta table in Microsoft Lakehouse

Question

I am currently working on a project where I am using Polars in a Fabric Notebook, and I would like to save this data into a Delta Table within a Microsoft Fabric Lakehouse. I was thinking of something like this:

df.write_delta(f"abfss://xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx@onelake.dfs.fabric.microsoft.com/xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx/Tables/xxx", 
               mode="append",
               storage_options={
                   "use_fabric_endpoint": "https://onelake.dfs.fabric.microsoft.com"
               })

However, I am encountering the following error:

OSError: Generic MicrosoftAzure error: Error performing token request: Error after 10 retries in 13.93100353s, max_retries:10, retry_timeout:180s, source:error sending request for url (http://xxx.xxx.xxx.xxx/metadata/identity/oauth2/token?api-version=2019-08-01&resource=https%3A%2F%2Fstorage.azure.com)

How to configure this correctly?

Thank you in advance for your help!

Best regards,

Mourad

Answer 1

If you're not using Spark or the local mount points, you're responsible for authenticating to OneLake.

And you don't appear to be authenticating to OneLake there. You would need to configure that library to use OAuth and a service principal to connect to OneLake.

The easiest thing is probably the local filesystem mount, which is pre-authenticated.

/lakehouse/default/Tables/...