Triggering a Databricks Delta Live Table from Azure Data Factory resets the whole tables. How do I disable that?
I have created a pipeline in Azure Data Factory that triggers a Delta Live Table in Azure Databricks through a Web activity mentioned here in the Microsoft documentation.
My problem is that when I trigger my DLT from ADF, it resets the whole tables, meaning that my data becomes unavailable during the pipeline execution. To be more clear, it has this additional step in the screenshot below:
However, when I run it directly from the Databricks UI, the tables will not get reset and the data is available during the execution of my pipeline. Here's how it looks like:
I would like to have the same behavior in ADF, as I have when trigger the pipeline directly from the Databricks UI. I don't want to have this additional "resetting tables" step in my DLT pipeline when I trigger it from ADF.
Anyone has any solution for this?
It looks like you have {"full_refresh": "true"} added to your web activity parameters - with this it will always do a full refresh. To avoid it, just pass the empty object (as {}) instead.