Search code examples
countcopyazure-data-factorypipeline

How to fail Copy activity in Azure Data Factory if the number of copied files is zero


I have a copy activity in ADF to copy files from a blob storage container to another container.

Files are filtered using "Wildcard file path" like "Energy*.zip".

I want the pipeline to report failure if the number of copied files is zero.

I cannot use "Get metadat" as there are millions of files in the source and it results in the below error:

The length of execution output is over limit (around 4MB currently)

Is there any way to get count of copied files and report failure if it is zero?

Pipeline has only one Copy activity and there are no other activities.


Solution

  • You can use Fail activity to fail the pipeline. Upon the success of copy activity, check the number of files written equals to Zero or not in if activity and use fail activity inside True activities of if.

    You can get the filesWritten count from the copy activity output.

    enter image description here

    Check this in if activity with below expression.

    @equals(activity('Copy data1').output.filesWritten,0)
    

    enter image description here

    Inside True activities of if activity, take a Fail activity and give any custom message and custom error code like below.

    enter image description here

    When the files written count by copy activity is Zero, this will fail the pipeline with the above error message.

    enter image description here