Search code examples
azure-data-factoryrestazure-data-lake-gen2

Download a file from REST API to Azure data lake using Azure Data Factory

I am new to the azure data factory, and I have followed tutorial online and learnt how to store a json response in a csv format by creating linked service and dataset for both data.

But what if the response is a file, like if I put that link(API) into web browser it will directly download the file, how can I download the file that being responded and store it into the Azure data Lake gen2? What dataset format should I use for the sink?

Thanks

Solution

  • To download the file using API, I would suggest you to use HTTP Connector in Azure Data Factory Linked Service, as shown below. As per my repro, I used copy activity to copy JSON file from HTTP request to Azure Data Lake Storage Gen2. enter image description here

    The data format for sink would be "DelimitedText" as your requirement is to convert the source JSON data into CSV. If you just want to store the exact data, then use JSON data format.

    Sink Dataset Details: enter image description here

    Here is the tutorial: https://www.cloudfronts.com/how-to-copy-data-from-rest-api-to-azure-data-lake-via-azure-data-factory/

    Should be fairly easy to follow, but if you have any questions be sure to reply me and ask away!