Search code examples
azurepipelineazure-data-factory

Is there a way to create dependencies between pipelines? (A single pipeline depend on +3 others)

I have 5 pipelines on my Azure Data Factory, each pipeline copy data to a different table. There is a dependency on some of this tables, table D & E depend on table A, B & C. Like in this example:

Table dependencies & Pipelines

What I'm doing to refresh all data is the following executing sequence:

Exec Pipeline to table A > Exec Pipeline to table B > Exec Pipeline to table C > Exec Pipeline to table D > Exec Pipeline to table E.

Pipelines Execution Order

I could execute Pipeline to table E before Pipeline to table D with no problem, but none of them can be executed before Pipelines for table A, B & C.

The idea I had to make this more organized and easier to schedule was change the pipeline D and add there 3 steps that will execute the Pipelines for table A, B & C. And on Pipeline to table E I added a step to execute the pipeline D. Like in the example:

Pipelines Idea

However, this would create a kind of dependency on Table E with Table D, which I don't want. If I need for any reason to update JUST table E, it won't be able because I would need to update table D first.

I wanted that both Pipelines to table D & E had a kind of validation if Pipelines to table A, B & C had run so then they can run.

Is there a way to make this dependencies more organized?

Solution

  • To make pipeline dependent on each other you can create a trigger and make those triggers dependent on other triggers.

    To create dependent pipelines, we can use Tumbling window trigger

    Create triggers for Pipeline to table A, B, C by

    Trigger >> New/Edit >> Choose Trigger >> New >> Type-Tumbling window >> Configure Properties >>Save

    To make Pipeline to table D, E dependent on Pipeline to table A, B, C

    Select Trigger > Advanced > New, then choose the trigger to rely on with the proper offset and size. This will create dependent on the trigger.

    Example

    enter image description here

    In above example Trigger 4 is created for pipeline 4 And in Dependencies added Trigger 1, 2, 3 which are created for pipeline 1, 2, 3 respectively.

    This dependent Tumbling window trigger execute only when another trigger inside the service has successfully been executed.