Search code examples
azureazure-data-factoryazure-eventhub

Moving already read Avro Files from One Directory to Another Using Azure DataFactory Pipeline

I have an Eventhub Capture which emits Avro Files Every 15 minutes. I can successfully read these files with an Azure Data Factory Data Flow however how can I move these files once I have finished reading them using an Azure DataFactory Pipeline?

There will be new files coming in and I don't want to move the new files only the already processed files to an archive folder.

I tried to do a GetMetadata Activity and then Delete but it doesn't seem to be able to ready Avro Files in a directory. Could be I'm missing something.

Thanks

Solution

  • Data flow can handle this organically:

    enter image description here