Search code examples
azure-batch

Unziping input files before running tasks

I want to be able to unzip an input file before running tasks to reduce data transfer in my batching process. To do this I added the following command line to my JobPreparationTask:

CommandLine = "/bin/bash -c 'DEBIAN_FRONTEND=noninteractive apt-get install -y unzip; unzip $AZ_BATCH_TASK_WORKING_DIR//myInput.zip'"

But instead of looking at my task working directory where I have my file (job/task/wd/myInput.zip) is looking at the job preparation folder and I get the following error:

"cannot find or open /mnt/batch/tasks/workitems/myJob/job-1/jobpreparation/wd//myInput.zip"

I have tried using other environmental paths but I always get similar errors. How can I access the job/task/wd to unzip the file?

Solution

  • I think its the env var which you are using is wrong, below are some recommendation as well answer to your current way here:

    • I assume that your zip file is part of resourceFiles in this case? But at what context?
    • Refer to resource files here: https://learn.microsoft.com/en-us/azure/batch/resource-files

    • In-particular at the part where it say: All types of tasks support resource files: tasks, start tasks, job preparation tasks, job release tasks, etc so at what point your resource files get uploaded under what context?

    • If upload is happening at the JobPrep and current task is referring to the JobPrep then that might explain this behavior.

    • if the Zip is part of the resoource files at task level then cmdline should be part of those tasks and the env var should be using on the current context.

    • If not: then make sure zip is part of resroucefiles with jobPrep and use the environment vars accordingly.

    • Regarding env var why are you not using : AZ_BATCH_JOB_PREP_WORKING_DIR or AZ_BATCH_JOB_PREP_DIR ? Please refer here : https://learn.microsoft.com/en-us/azure/batch/batch-compute-node-environment-variables#environment-variables which will allow access to your job prep dir level.

    Extra

    Idea share just so that you are aware and to keep in mind and only use if this ever fits your need. (Only use it if you have read the article and it actually fits your design scenario otherwise please ignore and above should resolve your issue) :)

    An application package is a .zip file that contains the application binaries and supporting files that are required for your tasks to run the application. Each application package represents a specific version of the application.

    You can specify application packages at the pool and task levels.

    Hope this fits your need. :) thanks