Search code examples
dataflowazure-data-factory

ADF IF Condition - Count source columns


ADF Copy activity - i have two types of flat files stored in the same directory, i am reading it with a wildcard *.txt

SOURCE type 1 has 18 columns, type 2 has 24 columns. the first 18 columns are identical, type 2 only has an additional 6 cols.

SINK azure sql table with 24 columns

Is it possible to use an IF condition to count the columns, and pass each source file into the correct copy activity - I would then have 2 copy activities (1 for each type of file).


Solution

  • You can use GetMetaData activity to get the structure of file and even count of columns. And then via If activity, split them for 2 different copy activities or else use dynamic mapping based on if condition leveraging only 1 copy activity

    enter image description here

    https://sqlitybi.com/dynamically-set-copy-activity-mappings-in-azure-data-factory-v2/