Azure Data Factory - Controlling order of transformations within a dataflow

Question

I'm trying to determine if there is some way to control the order of transformations WITHIN a dataflow in Azure Data Factory. I know that this can be done with multiple data flows in a pipeline but can you control the order of transformations within a data flow to ensure that one transformation does not process until another transformation has completed?

I have a very large data flow and I've noticed that as it's grown, the output can change between runs even though I've made no changes to the source data or the transformations in the data flow itself. I suspect this is because the multiple input streams run independently and a transformation is one stream that I reference in a second stream may or may not have completed before it is referenced in the second stream.

Here is a very simple example. I need some way to ensure that the 'AggregateTheData' transformation is completed before the 'SampleJoin' transformation is processed because the 'SampleJoin' transformation uses 'AggregateTheData' as one of its join inputs.

Answer 1

When you tried to use the output of any transformation, dataflow waits for the output of that transformation and then executes the current transformation.

I have tried same scenario as above.

sink1 creates file lookup2.csv and sink2 creates display2.csv files.

You can see I got the expected results in the display2.csv file.

Also, by the creation time of the files, it's clear that the above join transformation was executed after the aggregate transformation.

You can also set the sink write order like below to ensure the sink1 written first.