Search code examples
azureazure-databricksazure-data-factoryazure-data-lake

I want to copy files from File System to another FileSystem based on Country wise Tried with GetMetadata but ingesting only one file in the target

Files are in the form of Country wise in the FileSystem want to copy files as is in another Source System and then I need to copy them into adls So basically I need to use copy data twise if any code Suggestions that would be great Thank you in Advance ex: AU==>Filename1.csv Filename2.csv Filename3.csv UK==>Filename1.csv Filename2.csv EU==>Filename1.csv Filename2.csv source path

[1]: https://i.sstatic.net/55u82.png

target should be same but identically I need to copy in two locations by using single PL

Solution

  • You can use Preserve Hierarchy in the Copy behavior of copy activity to achieve your requirement. This takes the files from source directory and preserves the same hierarchy in the target. As you want to copy to 2 target locations, you need to use 2 copy activities.

    I have taken my source File structure as same as yours. First create a source delimited dataset and give the path only till Root container.

    Give this to copy activity and use wild card path in it like below to get your all files.

    enter image description here

    For sink, create another delimited dataset. Here for giving two target container names, use dataset parameter like below.

    enter image description here

    In copy activity sink, give your first target container name and select Preserve Hierarchy.

    enter image description here

    Create another copy activity like above. You can use Clone option for this.

    In this, give second target container name.

    enter image description here

    Execute this and you can get the desired result.

    Target 1:

    enter image description here

    Target2:

    enter image description here