Extracting All API Results into a Single CSV File Azure Data Factory

Question

I've got a JSON response from an API, and it's got a bunch of item information in a section called "Data." In this section, you'll find details like "ReleasePackageId," "Name," and "Version." Also, the response has some pagination stuff like "TotalCount," "PageIndex," and "PageSize."

What I'm trying to do is make a CSV file out of all this API data using Azure Data Factory. But here's the thing: the API gives the data in chunks, and the total number of items can be different each time (right now it's 88). So, I need to handle these chunks and put everything into one CSV.

In the world of ADF, I could use some help with the smartest way to go through these chunked API responses and grab the data. Any suggestions?

Answer 1

There are two ways to extract your API results into single CSV file by using ADF

1. If your API contains Pagination then you can use the Rest API connector to paginate on api and fetch and store the result in CSV file

• First create a linked service with REST API connector and set your API with pagination
• In source set the pagination according to your requirement.
• Set the sink as CSV file and set the correct mapping accorginly.
• Output

2. If you want to store the data from Http Connector in copy activity to single CSV file.

• First under the for each use copy activity and add source as HTTP connector along with your API and store the data into Json file.

• With proper mapping store it in json file in blob storage.

• Take another copy activity to merge that Json files to Single CSV file along with proper mapping.

• Output