Search code examples
azuresnowflake-cloud-data-platformazure-data-factoryazure-data-lakeazure-data-lake-gen2

Azure Data Factory V2 Copy Activity is not cleaning/deleting staged files/containers after copying data to the destination (Like: Snowflake)

Azure Data Factory V2 Copy Activity is not cleaning/deleting staged files/containers after copying data to the destination (Like: Snowflake).

Is this a bug? In my ADF, the container level is set for the staging area and I end up with multiple GUID containers. Some containers are even having metadata files.

According to Microsoft, all the temporary files/folders/containers created by Azure Data Factory in the Staging Area should be automatically deleted by the ADF after the Copy is successful.

Please guide.

Solution

  • According to Microsoft, all the temporary files/folders/containers created by Azure Data Factory in the Staging Area should be automatically deleted by the ADF after the Copy is successful.

    For stagging we have 2 options for linked service:

    1. Azure Blob Storage.
    2. Azure Data Lake Service.
    • Azure Blob Storage. -- If you selected Blob storage for staging it will delete or cleanup all folders which are created at the time of staging. -- As blob storage can't keep empty folders in it.

    Blob storage After stagging: enter image description here

    • Azure Data Lake Service. -- If you selected ADLS for staging it will just delete or cleanup the data file which is created at the time of staging. -- As you can keep empty folders in an ADLS.

    ADLS After stagging: enter image description here

    Container is not get deleted in either of the storage accounts.