Search code examples
u-sql

How to get all files from a folder and his subfolders?

Imagine this files paths:

  1. /root/subfolder/filename.csv
  2. /root/subfolder/subfolder2/filename2.csv

Can I extract data from "filename.csv" and "filename2.csv" without explicitly write their paths?

I want to do something like:

@var = EXTRACT column FROM "/root/{*}.csv" USING Extractors.Csv(skipFirstNRows:1);

Is it possible?

Solution

  • Unfortunately, this feature (a Kleene-* like recursive folder navigation) is not yet available in filesets, but is on our long term roadmap. Please file/upvote for this feature at http://aka.ms/adlfeedback.

    The current work-around is to have one step wild card EXTRACTs for each level you expect to encounter and then UNION ALL them together. E.g.,

     @d1 = EXTRACT ... FROM "/fixpath/{*}" USING ... ;
 @d2 = EXTRACT ... FROM "/fixpath/{*}/{*}" USING ...;
 @d3 = EXTRACT ... FROM "/fixpath/{*}/{*}/{*}" USING ...;
 ....

 @data = 
   SELECT * FROM @d1 UNION ALL SELECT * FROM @d2 UNION ALL SELECT * FROM @d3 UNION ALL ...;