Azure Data Factory Dataflow Add filter to Sink

Question

In Azure Data Factory in a dataflow we want to only modify sink if a condition is met ex ColumnA < 10

Is there any way we can filter (add where clause) to sink? FYI: The pipelines are dynamic, there are variable number of columns and variable number of primary key columns.

Answer 1

We use Sink transformation to store the data in destination after completing all other transformations. After Sink you cannot add any other transformation on the data.

As you can see in below the plus (+) sign to add transformations is only available after source and other transformation not after sink.

If you want to store the filtered data, then you need to use the Filter transformation before sink with appropriate condition e.g. Columnname < 10

To modify sink according to condition you need to use another data flow with same source and sink.

In Another dataflow select same source what is sink in previous dataflow the use filter condition to filter than data according to your need and then again in sink transformation select the same as source but in sink settings for Table action select Truncate or Overwrite this will overwrite or truncate the data before writing data in destination.