I have a bunch of Azure Batch Tasks (Windows) which are dependent on each other so that they are executed one after another. There is only one Job. Each Task requires all of the files generated in the previous Task (plus whatever files the previous Task got from its previous Task).
How should I transfer these files between the Tasks? My current solution is to execute a move command at the start of each commandLine for each Task. This move command moves all files from the previous Task's folder to the current Task's folder. This kinda works but doesn't seem right and I don't have any backup about the intermediary results.
I was thinking of setting all of the files as output for each Task (to Blob Storage) and setting them as input for the next Task but this doesn't work because I would have to know all files in advance to generate ResourceFile references for them. I don't know in advance which files will get generated. So my next best idea is to generate a ResourceFile reference for a single ZIP file which contains whatever contents the previous Task zipped into it. So the ZIP file's contents would change all the time but I can add it as input because I can create a ResourceFile reference to it (even if the file's contents change). But this seems rather cumbersome.
Other ideas?
P.S. This is related to my earlier question at Azure batch task dependencies: copy files from previous which mentions this same problem but asks a different question.
What you likely want is to mount a virtual file system on the pool allowing shared access. See https://learn.microsoft.com/en-us/azure/batch/virtual-file-mount