Search code examples
azure-data-factory

Populate a pipeline variable from a data source

I am loading some data from a REST source, which has a very simple structure:

[ {"id":"foo"}, {"id":"bar"}, ... ]

I am successfully able to use this source and save the results to the disk as a JSON file.

However, what I'd like to do is instead populate an array variable with these results. In a next step, I would then iterate through these IDs to do the rest of my work.

After quite a bit of googling, I have not found any documentation explaining how to do this, which has led me to start questioning if I'm even trying the right approach with this.

Is there a way to store the results of such an API call in a variable, or to otherwise iterate through these results?

Solution

  • After copy activity, use a lookup on the JSON. You can store the JSON array in an array variable by using lookup output.

    @activity('Lookup1').output.value

    enter image description here

    Output array in a variable:

    enter image description here