Search code examples
azure-data-factorymicrosoft-fabricdata-lakehouseonelake

Tables are going to files folder instead of tables in fabric lakehouse


I using the Azure Data Factory to ingest tables inside the Fabric Lakehouse, I did the ingestion from AWS S3 and SQL Server, it worked normally.

When I try to ingest from Azure Database for PostgreSQL and MySQL the tables is going to file folder instead of Table folder.

If I open the folder of each table file, I'll find in some tables, a parquet file and delta logs and other find just the delta logs, but if I try in the tables who has parquet files load inside the table folder, it works and the table appears.

Is there any configuration to do, so they automatically goes to table folder?

enter image description here enter image description here

Another thing, I said it worked with SQL Server and AWS S3, but I noticed they worked because the tables already exists, so I think just overwrite, but when the table doensn't exist, goes to file folder.

How the tables are been saving, each folder it's a table:

antoniofarias_6-1714747269733.png

How the files are been saving:

antoniofarias_1-1714744526661.png

Source:

antoniofarias_2-1714745442670.png

Sink:

antoniofarias_3-1714745619785.png

Dataset:

antoniofarias_4-1714745762021.png

Linked Service:

antoniofarias_5-1714746211408.png


Solution

  • After a lot investigation, the solution it's simple:

    When tables are ingested into a lakehouse, they initially land in an unconfirmed folder. If you try to access this folder by its table name, you might encounter a message stating that it's not possible to create a shortcut table, and that the table should be moved to a file folder to be kept inside the lakehouse. To resolve this, simply wait a few minutes, then refresh the lakehouse. The table should then appear in the table folder.

    If you move the unconfirmed tables to files folder, they won't go to table folder, so you will need to do this in manual form, one per one. It's happenning exatcly this, I was not waiting and moving all the table to files folder.

    enter image description here