Preventing Trigger Overwrites in ADF ARM Deployments

Question

I'm new to Azure, and I am deploying an Azure Data Factory (ADF) using an ARM template in Azure DevOps via a YAML file from DEV environment to QA environment. The problem is the triggers in DEV are overwriting the triggers in QA, which leads re-configuring the triggers in QA again.

My goal is to update pipelines, datasets, and other resources without modifying or overwriting existing triggers in the target environment.

This is the trigger in the ARMTemplateForFactory.json:

{
        "name": "[concat(parameters('factoryName'), '/MorningSchedule')]",
        "type": "Microsoft.DataFactory/factories/triggers",
        "apiVersion": "2018-06-01",
        "properties": {
            "annotations": [],
            "runtimeState": "Stopped",
            "pipelines": [
                {
                    "pipelineReference": {
                        "referenceName": "name_of_pipeline",
                        "type": "PipelineReference"
                    },
                    "parameters": {
                        "_parameter": "[parameters('name_of_parameter')]"
                    }
                }
            ],
            "type": "ScheduleTrigger",
            "typeProperties": {
                "recurrence": {
                    "frequency": "Day",
                    "interval": 1,
                    "startTime": "2023-11-30T11:30:00",
                    "endTime": "2026-12-01T09:30:00",
                    "timeZone": "Middle East Standard Time",
                    "schedule": {
                        "minutes": [
                            55
                        ],
                        "hours": [
                            9
                        ]
                    }
                }
            }
        },
        "dependsOn": [
            "[concat(variables('factoryId'), '_pipeline')]"
        ]
    }

And i was trying to play with the parametrization via the arm-template-parameters-definition.json but idk if that's will help. this is the code for triggers inside the arm-template-parameters-definition.json:

"Microsoft.DataFactory/factories/triggers": {
    "properties": {
        "pipelines": [
            {
                "parameters": {
                    "*": "="
                }
            },
            "pipelineReference.referenceName"
        ],
        "pipeline": {
            "parameters": {
                "*": "="
            }
        },
        "typeProperties": {
            "scope": "="
        }
    }
}

I've deleted the triggers manually from ARMTemplateForFactory.json but that deleted them from QA. It's deleting the triggers that is in QA env and replacing them with the triggers from the DEV env.

The YAML file is quite big, but i think this is the important part:

- task: AzureResourceManagerTemplateDeployment@3
  displayName: '...'
  inputs:
    deploymentScope: 'Resource Group'
    azureResourceManagerConnection: ...       
    subscriptionId: ...
    action: 'Create Or Update Resource Group'
    resourceGroupName: ...
    location: target_env_location
    templateLocation: 'Linked artifact'
    csmFile: Path to ARMTemplateForFactory.json
    csmParametersFile: Path to ARMTemplateParametersForFactory.json
    deploymentMode: 'Incremental'
    deploymentName: 'Name of Deployment'
    overrideParameters: 'Some Parameters'

I'm looking to see if there's a solution that will allow me to exclude triggers from the deployment process.

Answer 1

You are correct to play with the ARM template ARMTemplateForFactory.json to customize your deployment, however, you should NOT directly modify (commit & push) the ARM templates in the publish branch of Azure Repos.

Please note that the ARM templates in your repo are extracted from your ADF resources of DEV environment, therefore it uses the ADF resources' default values for deployment across environments and the triggers properties of your QA environment are overwritten by such a deployment based on DEV

Here is a sample original ARM template ARMTemplateForFactory.json, where we can see the properties for one of the triggers resource are hardcoded.

{
    ...
    "resources": [
        {
        {
            "name": "[concat(parameters('factoryName'), '/MorningSchedule')]",
            "type": "Microsoft.DataFactory/factories/triggers",
            "apiVersion": "2018-06-01",
            "properties": {
                "annotations": [],
                "runtimeState": "Stopped",
                "pipelines": [
                    {
                        "pipelineReference": {
                            "referenceName": "_pipeline",
                            "type": "PipelineReference"
                        },
                        "parameters": {}
                    }
                ],
                "type": "ScheduleTrigger",
                "typeProperties": {
                    "recurrence": {
                        "frequency": "Day",
                        "interval": 1,
                        "startTime": "2023-11-30T11:30:00Z",
                        "timeZone": "UTC",
                        "schedule": {
                            "minutes": [
                                55
                            ],
                            "hours": [
                                9
                            ]
                        }
                    }
                }
            },
            "dependsOn": [
                "[concat(variables('factoryId'), '/pipelines/_pipeline')]"
            ]
        },
    ...
    ]
}

For your expectation to exclude all the existing triggers in DEV and leave triggers in QA unchanged, you may consider parameterizing those default property values for triggers resources, so that you can override them by supplying the expected environment-specific values during deployment. See Custom parameters in a Resource Manager template - Azure Data Factory | Microsoft Learn.

Here are the brief steps for your reference.

1. Navigate to Source control -> ARM template -> ARM template configuration -> Edit parameter configuration;

2. I would like to parameterize the recurrence.interval and recurrence.frequency property for one of my triggers resource, so I added "*": "=" in node of properties.typeProperties.recurrence of "Microsoft.DataFactory/factories/triggers";

3. Export the ARM template and verify the parameters are populated in ARMTemplateForFactory.json;

   

   

4. Publish the templates into the publish branch of your Git provider(Azure Repos);

   

5. Override the parameter values in deployment pipeline.

variables:
  ARMSvcCnn: ARMSvcCnnWIFAutoSub1
  QA.resourceGroup.name: rg-azdatafactory
  QA.location: southeastasia
  DEV: xxx-instance1
  QA: xxx-instance2

pool:
  vmImage: ubuntu-latest

steps:
- task: AzureResourceManagerTemplateDeployment@3
  inputs:
    deploymentScope: 'Resource Group'
    azureResourceManagerConnection: $(ARMSvcCnn)
    subscriptionId: $(subscription.id)
    action: 'Create Or Update Resource Group'
    resourceGroupName: $(QA.resourceGroup.name)
    location: $(QA.location)
    templateLocation: 'Linked artifact'
    csmFile: '$(Build.SourcesDirectory)/$(DEV)/ARMTemplateForFactory.json'
    csmParametersFile: '$(Build.SourcesDirectory)/$(DEV)/ARMTemplateParametersForFactory.json'
    overrideParameters: >
      -factoryName "$(QA)"
      -MorningSchedule_properties_typeProperties_recurrence_frequency "Week"
      -MorningSchedule_properties_typeProperties_recurrence_interval "3"

With those being said, please keep in mind:

1. All those templates are generated by the operations in ADF. DO NOT manually push commits to modify these templates in Azure Repos.

2. The only change from the Azure DevOps side is to modify the deployment pipeline to override customer parameters. Ensure the parameters are correctly added in the ARM templates - In ADF, Export them for verification prior to publishing to Azure DevOps.