Search code examples
azureazure-data-factoryazure-data-lakeazure-data-lake-gen2

ADF Delete Activity not deleting Folders

I have a pipeline to delete some folders from ADLS. My folder structure is as below.

/raw/MainFolder/SubfolderA/20230430/File.parquet
/raw/MainFolder/SubfolderA/20230415/File.parquet
/raw/MainFolder/SubfolderA/20230410/File.parquet

/raw/MainFolder/SubfolderB/20230430/File.parquet
/raw/MainFolder/SubfolderB/20230420/File.parquet
/raw/MainFolder/SubfolderB/20230405/File.parquet

The pipeline is currently deleting the File.parquet under the date folder. But I need it to delete the whole folder named with date (20230430).

Currently the pipeline is deleting the parquet files when they exist and erroring out when the folder is empty. I am not passing the parquet file name to my pipeline. (Just the folder path) I have recursively enabled in the delete activity.

Error: Failed to execute delete activity with data source 'AzureBlobStorage' and error 'The required Blob is missing. Folder path: raw/MainFolder/SubfolderA/20230430/.'.

How do I get the delete activity to delete the folder itself and not just the files in it?

I am very new to Azure data factory. Appreciate your help.

Pipeline is deleting the files but not the folders

Solution

  • Error: Failed to execute delete activity with data source 'AzureBlobStorage' and error 'The required Blob is missing. Folder path: raw/MainFolder/SubfolderA/20230430/.'.

    The above error will cause because of For ADLS path you are using blob storage linked service.

    In blob storage you can't delete empty directory/folder .

    I also got the same error when i used blob storage linked service.

    enter image description here

    To resolve this use ADLS linked service, and to delete particular folder from ADLS irrespective of file contain in it.

    • First take a delete activity and create a source dataset with ADLS linked service for it with just folder name. enter image description here Source dataset settings: folder name is MainFolder/SubfolderA/20230430 enter image description here

    This will delete respective folder.

    before deleting folder: enter image description here After deleting folder 20230430: enter image description here

    Dataset.json

    {
    "name": "Binary1",
    "properties": {
        "linkedServiceName": {
            "referenceName": "AzureDataLakeStorage1",
            "type": "LinkedServiceReference"
        },
        "annotations": [],
        "type": "Binary",
        "typeProperties": {
            "location": {
                "type": "AzureBlobFSLocation",
                "folderPath": "MainFolder/SubfolderA/20230430",
                "fileSystem": "raw"
            }
        }
    }
}

    Pipeline.json

    {
    "name": "pipeline1",
    "properties": {
        "activities": [
            {
                "name": "Delete1",
                "type": "Delete",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "dataset": {
                        "referenceName": "Binary1",
                        "type": "DatasetReference"
                    },
                    "logStorageSettings": {
                        "linkedServiceName": {
                            "referenceName": "AzureDataLakeStorage1",
                            "type": "LinkedServiceReference"
                        },
                        "path": "raw/logs"
                    },
                    "enableLogging": true,
                    "storeSettings": {
                        "type": "AzureBlobFSReadSettings",
                        "recursive": true,
                        "enablePartitionDiscovery": false
                    }
                }
            }
        ],
        "annotations": []
    }
}