Search code examples
azureazure-data-factory

Delete folders specific folder in adf

I have the follwing set of activities ( see below image) which deletes the folders which are older than 5 days. My path looks like this bronze/D365/Snapshot/salestable/"todaydate"/*.parquet. Below pipeline deletes folders from salestable folder.

enter image description here

In the get metadataactivity I have following settings: enter image description here

where the folder_path is only pointing( hardcoded) SalesTable.

My goal is to make this dynamic and pass all the table names as I dont have only salestable there.

I tired to but the entire pipeline under ForEach loop and before for each loop another Getmetada activty which gives all the table names. Like this:

enter image description here

my problem is that I dont know how to pass the value from the Getmetadata_Of_ALL inside ForEach activity for get metadata activity dataset. So dataset filepath will become dynamic

Solution

    • You can use get metadata on bronze/D365/Snapshot/ path as use field list as child items to get list of folders inside this path (since you have multiple folders along with salesTable).

    • The following is the demo of the same. I have folder structure as data/a/b inside which I have 3 folders (like salesTable and inside which I have todays date folder). I have used get Metadata on these and got the result as shown below:

    enter image description here

    • Now iterate through each of these folders. Inside for each, use another get metadata activity with dataset JSON as below:
    {
    "name": "csv2",
    "properties": {
        "linkedServiceName": {
            "referenceName": "adls1",
            "type": "LinkedServiceReference"
        },
        "parameters": {
            "folder_name": {
                "type": "string"
            }
        },
        "annotations": [],
        "type": "DelimitedText",
        "typeProperties": {
            "location": {
                "type": "AzureBlobFSLocation",
                "folderPath": {
                    "value": "a/b/@{dataset().folder_name}",
                    "type": "Expression"
                },
                "fileSystem": "data"
            },
            "columnDelimiter": ",",
            "escapeChar": "\\",
            "quoteChar": "\""
        },
        "schema": []
    }
}
    • Pass the value for folder_name parameter from the get metadata activity as @item().name.

    enter image description here

    • This would give the contents of each of the folders returned in the first get metadata activity which can be further used.

    enter image description here