Search code examples
azureazure-batch

Are task logs deleted when the node is deleted due to autoscaling in Azure Batch service?

I am trying to access the logs (stdout.txt and stderr.txt) of the tasks run.

As soon as the task is completed I lose access to those files. Looking at the browser console, if I search a file in the path field I get a 404 error, NodeNotFound, The specified node does not exist.

Is it caused because the node is deleted when the task completes? Is there a way to retain those files after the node is deleted?

Thanks in advance.

P.S.: Also asked in the Microsoft Learn platform (https://learn.microsoft.com/en-us/answers/questions/1820060/are-task-logs-deleted-when-the-node-is-deleted-due).

Solution

  • Are task logs deleted when the node is deleted due to autoscaling in Azure Batch service?

    The above query is solved in MS-Q&A by Prrudram-MSFT.

    Posting it an answer so that it will help community to find better solution.

    Is it caused because the node is deleted when the task completes? Is there a way to retain those files after the node is deleted?

    • When a node is removed from the pool, all the files on that node are deleted. Therefore, it's important to move the output of your task to a durable storage location before the task completes. Likewise, if a task fails, you should transfer any logs needed for diagnosing the failure to a durable storage location.
    • Batch provides integrated support for Azure Storage to upload data via OutputFiles. It also supports various shared file systems, or you can choose to upload the data yourself within your tasks.

    Regarding your question, you can save the log files using Task output files. Task output files provide a convenient way to save log files to Azure Storage. By specifying the output files in the task definition, Batch will automatically upload the files to your storage account when the task completes.

    Reference: Persist job and task output to a data store - Azure Batch | Microsoft Learn