Filtering rows in Azure Synapse/Data Factory dataflow

Question

I have been stuck for a bit trying to sort this out and for some reason it is not working.

I am reading a parquet file on a Pipeline in Synapse, after that I am selecting the columns I want to keep and before doing the sink, I want to add a filter (WHERE clause in SQL) to only keep specific rows.Here is the set up

Set up

Currently if I do a Data preview of the SelectColumns output I get data just fine.

data

And here is the definition of my filter

filter definition

But if I check the Data Preview for the filter task, it says no doutput data.

Can someone help me finding out whats going on please? I even tried a simpler filter SCTSTCO=='ABCD' just for testing and the same.

Am I just being too dumb?

Thanks a lot for the help

Answer 1

The above expression is working fine for me.

The expression CTSOCO=='ABCD' || CTSOCO=='ACRF' also working fine in the filter transformation.

If your data preview is resulting no rows, you might need to refresh the dataflow debug here.