Azure batch task dependencies: copy files from previous

I have a Azure Batch scenario where I have a chain of Tasks which are run after each other. Dependencies are set correctly so they run nicely after each other.

However I need to copy all files from the previous Task's folder to the new Task's folder before execution. I do not know in advance how many and what files there will be so I just want to copy everything. I could not find a way to accomplish this with the Batch client library (https://learn.microsoft.com/en-us/dotnet/api/overview/azure/batch?view=azure-dotnet).

As a workaround I tried adding a simple copy task to the .bat file which is executed with commandline but for some reason it only copies some of the files. In one task there are a few hundred files to copy and it varies a few % how big portion it copies before it stops copying (with no errors). This is my copy command: $"cmd /c xcopy /E /F /Y %AZ_BATCH_TASK_WORKING_DIR%\\..\\..\\{previousTaskId}\\wd %AZ_BATCH_TASK_WORKING_DIR%". Everything works correctly if performed directly from the VM.

Tested hypothesis:

Copying overwrites the .bat file which executes the actual processing. This in turn breaks the copying. I've now ruled out this problem (each task has a differently named .bat file)
Copying is done for some reason in parallel. I added timestamp echos to the bats and there is no parallelism so this can't be the reason. Also tried adding sleep 10 before the xcopy but didn't make any difference.
xcopy wouldn't see all the files for some reason. Added a dir command to see what files there are and it sees only the same files which xcopy copies.
user access issues. Doesn't make sense as some files are copied succesfully and there are no errors.

Any ideas? This sounds like a trivial scenario but I just couldn't figure out how to do this.

Solution

It turned out that the problem was in the previous Task: it launched a process which started generating the files in the background and returned control immediately. Therefore the Batch engine thought the Task had finished and continued to the next Task which was first copying the files generated by the previous Task.

My hypothesis about parallelism was therefore partially true although it wasn't visible with echoing timestamps (first Task said it finished before second Task said it started). The experiment with sleep would've revealed the problem but I either used too short sleep delay or somehow read the results wrong.

Because I can't control how the first Task launches the process I now added some Windows Batch script to poll tasklist about when the process ends and it solved the problem.