I have the following code:
@dlt.table(
name="ingested_data",
comment="Ingest the table",
table_properties={
"quality": "raw", "name": "property_name"
}
)
I am confused what the table_properties
dictionary does in practice? I have seen no effect of the key-value pairs. Here is the list of the accepted keys.
Table properties are used for two things:
pipelines.reset.allowed
defines if it's allowed to perform a reset on the given table (it's good idea to disable it on the table that holds data ingested from Kafka), or pipelines.autoOptimize.zOrderCols
defines a list of columns that will be used in OPTIMIZE ZORDER BY <cols>
.raw
in your example`), tags, etc. that could be used when searching for tables in UI.1st kind of usage happens often in practice, while 2nd is optional.