Search code examples
azureazure-data-factory

Azure Data Factory: Is it possible to use a lookup activity or get metadata activity to query a Delta table

I have a delta table located in an ADLS Gen 2 whose contents I need to retrieve using a lookup activity. I would like to be able to do this without connectivity to databricks as I have built this using an AZureDeltaDatabricks dataset. Is it possible to achieve this with an ADLS dataset instead?

Solution

  • Delta files are actually types of Parquet files, so you can use the Parquet file source to query them.

    1. create a linked service of type Azure Data Lake Storage Gen2 to your storage account
    2. create a dataset of type Azure Data Lake Storage Gen2 and file type Parquet to the actual file or its folder
    3. use this dataset in your lookup as the source

    Please note that although this will work for simple files, I'm not sure what will happen if you delete\update data within the delta table, since Parquet source will not know how to read Delta change log. If you need support for this, you may require databricks or Synapse Serverless pool to access the Delta file.