Search code examples
azureazure-data-factoryazure-bicep

Missing parameter definition when deploying adf with bicep


I am deploying an adf with bicep similar to this https://learn.microsoft.com/en-us/azure/data-factory/solution-template-move-files template from microsoft. It worked nicely when I ran the pipeline by editing in the UI. I then exported the arm template from the datafactory and have converted it into a bicep template with the same exact parameters. However I get this error message when deploying:

Status Message: Missing parameter definition for SourceStore_Location,SourceStore_Directory

Should I not be able to reference the parameters like this? Have anyone done something similar?

resource dataFactoryPipeline 'Microsoft.DataFactory/factories/pipelines@2018-06-01' = {
  parent: dataFactory
  name: pipelineName
  properties: {
       parameters: {
      SourceStoreLocation : {
        type: 'String'
        defaultValue: 'xxxx\\xxxx\\xxxx\\xxxx'
      }
      DestinationStoreLocation : {
        type: 'String'
        defaultValue: 'xxxx\\xxxx'
      }
      SourceStoreDirectory : {
        type: 'String'
        defaultValue: 'xxxx'
      }
      DestinationStoreDirectory : {
        type: 'String'
        defaultValue: 'xxxx'
      }
    }
    activities: [
      {
        name : 'GetFileList'
        type: 'GetMetadata'
        typeProperties: {
          dataset: {
            referenceName: sourceDataSetName
            type: 'DatasetReference'
            parameters: {
              FolderPath: {
                value : '@pipeline().parameters.SourceStore_Location'
                type: 'Expression'
              }
              Directory: {
                value: '@pipeline().parameters.SourceStore_Directory'
                type: 'Expression'
              }
            }
          }
          fieldList: ['childItems']
          storeSettings: {
            type: 'FileServerReadSettings'
            recursive: true
          }
          formatSettings: {
            type: 'DelimitedTextReadSettings'
          }
        }

      }
      {
        name : 'FilterFiles'
        type: 'Filter'
        dependsOn: [
          {
            activity: 'GetFileList'
            dependencyConditions: [
              'Succeeded'
            ]
          }
        ]
        typeProperties: {
         
         
        items:{
          value: '@activity(\'GetFileList\').output.childItems'
          type: 'Expression'
        }
        condition: {
          value: '@equals(item().type, \'File\')'
          type: 'Expression'
        }
        }

      }
      {
        name : 'ForEachFile'
        type: 'ForEach'
        dependsOn: [
          {
            activity: 'FilterFiles'
            dependencyConditions: [
              'Succeeded'
            ]
          }
        ]
        typeProperties: {
          items: { 
          value : '@activity(\'FilterFiles\').output.value'
          type : 'Expression'
        }
        activities: [
          {
            name : 'CopyFile'
            type: 'Copy'
            typeProperties: {
              source: {
                type: 'DelimitedTextSource' 
                storeSettings: {
                  type: 'FileServerReadSettings'
                  recursive: true 
                }
                formatSettings: {
                  type: 'DelimitedTextReadSettings'
                }
              }
              sink: {
                type: 'DelimitedTextSink'
                storeSettings: {
                  type: 'AzureFileStorageWriteSettings'
                }
                formatSettings: {
                  type: 'DelimitedTextWriteSettings'
                  quoteAllText: true 
                  fileExtension: '.txt'
                }

              }
            }
            inputs: [
              {
                 referenceName: sourceDataSetFileName
                 type: 'DatasetReference'
                 parameters: {
                  FolderPath: {
                    value : '@pipeline().parameters.SourceStore_Location'
                    type: 'Expression'
                  }
                  Directory: {
                    value: '@pipeline().parameters.SourceStore_Directory'
                    type: 'Expression'
                  }
                  FileName: {
                    value: '@item().name'
                    type: 'Expression'
                  }
                 }     
              }
            ]
            outputs: [
              {
                referenceName: sinkDataSetName
                type: 'DatasetReference'
                parameters: {
                  FolderPath: {
                    value : '@pipeline().parameters.SourceStore_Location'
                    type: 'Expression'
                  }
                  Directory: {
                    value: '@pipeline().parameters.SourceStore_Directory'
                    type: 'Expression'
                  }
                  FileName: {
                    value: '@item().name'
                    type: 'Expression'
                  }
                }
              }
            ]
          }
        { 
          name : 'DeleteFiles'
          type: 'Delete'
          dependsOn: [
            {
              activity: 'CopyFile'
              dependencyConditions: [
                'Succeeded'
              ]
            }
          ]
          typeProperties: {
            dataset: {
              referenceName: sourceDataSetFileName
              type: 'DatasetReference'
              parameters: {
                FolderPath: {
                  value : '@pipeline().parameters.SourceStore_Location'
                  type: 'Expression'
                }
                Directory: {
                  value: '@pipeline().parameters.SourceStore_Directory'
                  type: 'Expression'
                }
                FileName: {
                  value: '@item().name'
                  type: 'Expression'
                }
              
              }
              
            }
            storeSettings: {
              type: 'FileServerReadSettings'
              recursive: true 
            }
          }

        }
        ]
        
         
        }

      }
      
    ]
 
  }
  dependsOn: [
    sinkDataSet
    sourceDataSet
    sourceDataSetFile
  ]
}

Solution

  • Strangely it seems like it may have fixed itself? I hardcoded the parameters in the different activities and then added the expressions one by one, deploying the template to weed out where it stopped. In the end I added them all and then suddenly it worked