Search code examples
copyazure-data-factorysink

Azure Data Factory: Copy file mentioned in JSON

I have a JSON where one the parts is a filename.

{
    "dial_target_file_row_count": 2440,
    "dial_target_file_endpoint": "consumerzone@dialconsumingp01dls",
    "dial_target_file_full_path": "abfss://*****/data/snapshotdate=2023-03-25",
    "dial_metadata_file_location": "abfss://*****/metadata/",
    "dial_metadata_file_name": "dsapp_goldenDataset_2023-03-25_20230326075008_metadata.json",
}

Now I want to copy the file 'dail_target_file_full_path' onto my blob storage. How do I define the source and sink (the sink name should be a part of the original filename).

I don't see any option to have a dynamic filename

Solution

  • Add a lookup activity to your pipeline to get the content of the JSON with the file details. If this is one file, check "first row only". Add a copy data activity, and create a dataset with parameters: dataset

    In the copy data source, you can now add values for this parameters, which should be the result of the first JSON: source Here are the values I provided in dynamic content for this parameters:

    @activity('Lookup1').output.firstRow.dial_metadata_file_location
@activity('Lookup1').output.firstRow.dial_metadata_file_name

    You will need to get just a part of the file path for the folder (without the url part)