Search code examples
azure-data-factory

How to change the type, to an array, of an item I'm passing to identify key column(s) in Data Factory?


I have a lookup where I'm gathering tables from a couple of schemas. In that same lookup, I've got a column that identifies key columns needed for the Upsert operation on the Sink. I'm then doing a "For Each" and a copy activity inside.

Here's how I've set up things in there for my sink: enter image description here

I tried running this, however, key columns needs to be in type Array. How would I accomplish changing the type here?

Sample output from lookup:

+--------------+------------+-------------+
| table_schema | table_name | key_columns |
+--------------+------------+-------------+
| schema1      | table1     | key1        |
+--------------+------------+-------------+
| schema2      | table2     | key2, key3  |
+--------------+------------+-------------+

Solution

  • When you are upserting data in SQL and passing key columns, it should be array format and from you sample data it is in the comma separate column names to pass them as an array you need to use dynamic expression as @split(item().key_columns,',' ) .

    It will take string value of key_columns value and split it where it found a comma and create array of that values.

    See below images for reference:

    My lookup Output:

    enter image description here

    Copy activity sink settings and output:

    enter image description here

    enter image description here