Search code examples
azureetlazure-data-factory

How to copy the data from Append variable activity to a csv file using Azure Data Factory

How to copy the data from append variable activity to a csv file using Azure Data Factory

I have array of file names stored in append variable activity. I want to store all these files names inside a .CSV file in the data lake location.

For more info refer this how to compare the file names that are inside a folder (Datalake) using ADF

Solution
    • In this repro, Variable V1 (array type) is taken with values as in below image.

    enter image description here

    • New variable v2 of string type is taken and value is given as @join(variables('V1'),decodeUriComponent('%0A')) This step is done to join all the strings of the array using \n (line feed). enter image description here

    • Then Copy activity is taken and dummy source dataset with one row is taken. enter image description here

    • In Source, +New is selected and value is given as dynamic content @varaiables('v2').

    enter image description here

    • Sink dataset is created for CSV file.
    • In Mapping, import schemas is clicked and other than col1 , all other columns are deleted.

    enter image description here

    • Then pipeline is debugged, and values got loaded in csv file. enter image description here

    Edited

    • Variable v2 is storing all the missing file names. (False activity of IF condition)
    • After for-each, Set variable is added and variable v3 (string type) is set as @join(variables('v2'),decodeUriComponent('%0A'))

    enter image description here

    • Then, in copy activity, +New column is added in source enter image description here