Move Entire Azure Data Lake Folders using Data Factory?

Question

I'm currently using Azure Data Factory to load flat file data from our Gen 2 data lake into Synapse database tables. Unfortunately, we receive (many) thousands of files into timestamped folders for each feed. I'm currently using Synapse external tables to copy this data into standard heap tables.

Since each folder contains so many files, I'd like to move (or Copy/Delete) the entire folder (after processing) somewhere else in the lake. Is there some practical way to do that with Azure Data Factory?

Answer 1

Yes, you can use copy activity with a wild card. I tried to reproduce the same in my environment and I got the below results:

First, add source dataset and select wildcard with folder name. In my scenario, I have a folder name pool.

Then select sink dataset with file path

The pipeline run is successful. It transferred the file from one location to another location with the required name. Look at the following image for reference.