Search code examples
azure-data-factoryazure-data-lake-gen2

Azure Data Factory Read Csv Files from multiple portioned folders

I am trying to read csv file from partitioned folders using dataflow activity ((yyyy/mm/dd/timestamp). how to read the csv files dynamically?

enter image description here

enter image description here

enter image description here

How to read single short for all the sub folders? above folder structure before start YYYY (year), the table name as a folder path.

eg : test/dwh/tablename/2023/01/15/100543/files.csv

enter image description here

Solution

  • You can use Wildcard paths in source options in dataflow as below. test/dwh/tablename/**/**/**/**/*.csv

    enter image description here

    Here, you can give the Partition root path, that tablename in your case.

    In my case following is the path sampletable/2022/01/12/102344/part1.csv In each csv files there are 6627 records.

    And this is the number of rows written.

    enter image description here.