Search code examples
databricksazure-databricksdelta-lakeazure-data-lake-gen2

How to write to a folder in a Azure Data Lake container using Delta?

How do I write to a folder in an Azure Data Lake container using Delta?

When I run:

    write_mode = 'overwrite'
    write_format = 'delta'
    save_path = '/mnt/container-name/folder-name'
     
    df.write \
        .mode(write_mode) \
        .format(write_format) \
        .save(save_path)

I get an incompatible format error. This does not happen when I write directly to the container ('/mnt/container-name').

I have also tried writing using the SQL commands but I get the same error.

What am I doing wrong?

Solution

  • It turns out that you cannot save to a folder within a delta table. The mistake that I made was I saved the delta table directly to the container and then tried to save another delta table to a folder within this container.

    When I saved to a folder within a container that only had folders in it, I did not get this problem.