Search code examples
azure-data-factoryazure-synapse

Disable activity within pipeline in Azure Synapse/Data Factory

Pipeline in question

We are building a Data Warehouse on Synapse using ETL. We need a way to stop and start activities automatically in case of issues or during deployments, as the current manual disable/enable option is impractical for production.

Is there an alternative method to automate this?

Would a PowerShell command be available to enable/disable activities?

We also considered using an If condition with a control table to check each activity, but this seems too complex.

Solution

  • Is there an alternative method to automate this? Would a PowerShell command be available to enable/disable activities?

    Agreed with @Joel Cochran and AFAIK, there is no other Dynamic or automated way like PowerShell command to enable/disable activities within the Azure data factory Pipeline. You need to do it manually only from the Azure Data Factory Portal.

    So, the possible solution is with the help of metadata driven table which help to decide if the particular activity should be enabled or disabled with help of lookup activity and If Activity of azure data factory

    • Create a table on SQL where you specified which activity need to be enabled or which need to be disabled. as below:
    ActivityName IsEnabled
    Activity1 1
    Activity2 0
    • Use lookup activity to check the IsEnabled value from the above table with below query and pass it to If activity.
    SELECT IsEnabled FROM ActivityControl WHERE ActivityName = 'Activity1';
    • Check the value for IsEnabled is true or not if activity and place respective activity in true condition so if vale is 1 it will execute the activity.
    @equals(activity('LookupActivity').output.firstRow.IsEnabled, 1)

    Repeat this process for each activity to automatically enabled or disabled.