Search code examples
azureazure-data-factoryazure-data-lake

Azure ADF how to ensure that the same files that are copied, are also deleted?

Using Azure ADF and currently my setup is as follows:

Event based triggerd on a input BLOB on file upload. File upload triggers a copy activity to output BLOB, and this action is followed by a delete operation on the input BLOB. The input BLOB can take 1 or many files at once (not sure how often the file is scanned/how quickly the event triggers the pipeline). Reading up on the delete function documentation it says:

Make sure you are not deleting files that are being written at the same time.

Would my current setup delete files that are being written?

Event based trigger on file upload >> Write from input Blob to Output Blob >> Delete input Blob

I've made an alternative solution which does a get metadata activity based on event in the beginning of the pipeline, and then does a for loop which deletes the files at the end, not sure if this is necessary though. Would my original solution suffice in an unlikely event where I'm receiving files every 15seconds or so?

Also while I'm at it, in a get metadata activity how can I get the actual path to the file, not just the file name?

Thank you for the help.

Solution

  • Delete Active says:

    • Make sure you are not deleting files that are being written at the same time.

    Your settings are:

    • Event based trigger on file upload >> Write from input Blob to Output Blob >> Delete input Blob

    Only after the active Write from input Blob to Output Blob finished(the deleting files are not being written), then the Delete input Blob can works.

    Your questions: Would my current setup delete files that are being written?

    So did you test these steps? You must test by yourself and you will get the answer.

    Please notice:

    Delete activity does not support deleting list of folders described by wildcard.

    Any other suggestions:

    You don't need to using delete actives to delete the input blob after Write from input Blob to Output Blob finished.

    You can learn from Data flow, it's Source settings support delete the source file(input blob) after the copy active completed. enter image description here

    Hope this helps.