Search code examples
azure-synapseazure-data-factory

Requiremet of Error logging mail(mail should contain all the missing file details). in the ADF flow

the reqirement is simple , i have a folder having 4 txt files(1.txt,2.txt,3.txt,4.txt) . the Flow is controlled by a parameter called all or some which is of string type. This is the adf pipeline, with the activities

If i select all in the parameter, all 4 file should be processed. the requirement starts here >>

  1. IF any file is missing from the folder(for ex 2.txt and 3.txt is not present and i selected ALL in the parameter) , i need a mail saying file is 2.txt and 3.txt is missing.

  2. If i select some in the parameter, for ex 1.txt and 4.txt and if any of the file is missing 1.txt and 4.txt is missing(for example 1.txt is missing) , i need a mail with the missing file name(i.e 1.txt in our case).

Solution

  • capture missing file details in one variable

    I tried to repro this capturing missing files using azure data factory. Below is the approach.

    • Take a parameter of array type in the pipeline. During runtime, you can give the list of file names in a folder to be processed in this array parameter. enter image description here

    • Take a get metadata activity and add a dataset in the activity. Click +New in the field list and select child items as an argument.

    enter image description here

    • Take a filter activity and give the array parameter value in items and write the condition to filter the missing files in condition box

    item: @pipeline().parameters.AllorSome

    condition: @not(contains(string(activity('Get Metadata1').output.childItems),item()))

    • I tried to run this pipeline. During run time, four file names are given to the array parameter. enter image description here

    • Get Metadata activity output has three file names.

    enter image description here

    • Parameter has 4 filenames and Get meta data activity has 3 filenames. Missing file names are to be filtered out.
    • Output of filter activity:

    enter image description here