can get metadata activity fetch data recursively at ADF pipeline?

Question

i am currently mainly doing the practicing in ADF pipeline and I realised if i use getmetadata activity and select child items it won't go deep inside of the folder of the folder, which is not doing recursively, is anyway can achieve it? i found if directly using copy activity and select wildcard path can active recursive and allow all the files no matter in which folder can copy to the destination, but if i need additional function like filter by last modify date or filter by other condition I still need use get metadata and filter activity

Answer 1

When you get metadata from file stores and configure modifiedDatetimeStart or modifiedDatetimeEnd, the childItems in the output includes only files in the specified path that have a last modified time within the specified range. Items in subfolders are not included.

The output of getmetadata includes only files in the specified path to get nested values you need to iterate on output directories of getmetadata values. using foreach loop this way you can go up to 1 level deep.

@Richard Swinbank here Get Metadata recursively in Azure Data Factory document discussed the Workaround for same situation.

An alternative to attempting a direct recursive traversal is to take an iterative approach, using a queue implemented in ADF as an Array variable. You can follow above documentation to create this iterative approach.