Search code examples
databricksdelta-live-tables

Effect of "table_properties" property for Delta Live Tables


I have the following code:

@dlt.table(
    name="ingested_data",
    comment="Ingest the table",
    table_properties={
        "quality": "raw", "name": "property_name"
    }
)

I am confused what the table_properties dictionary does in practice? I have seen no effect of the key-value pairs. Here is the list of the accepted keys.


Solution

  • Table properties are used for two things:

    1. Defining specific configurations that will be used by the underlying engine - they are listed in documentation. Like, pipelines.reset.allowed defines if it's allowed to perform a reset on the given table (it's good idea to disable it on the table that holds data ingested from Kafka), or pipelines.autoOptimize.zOrderCols defines a list of columns that will be used in OPTIMIZE ZORDER BY <cols>.
    2. Allow to specify arbitrary table properties/metadata, like, data quality level (raw in your example`), tags, etc. that could be used when searching for tables in UI.

    1st kind of usage happens often in practice, while 2nd is optional.