How to use Azure Batch in an event based design and terminate/cleanup finished jobs

Using Azure Batch, my project adds jobs to a pool using an event based design with functions and queues. When the job is finished, it is still "active", even though all tasks are completed.

A (single using an app service plan) function is triggered on a timer which reads an X amount of messages from the queue. The function:

Creates a pool (if it does not exist)
Creates a job
Adds the tasks to that job

That works good. However, once the tasks have finished, the job status remains active, even though all tasks have finished. I want the jobs to terminate/cleanup/set the status to "completed".

And I want my functions to be short-lived and do not want any statefullness. So I am not using foreach (CloudTask task in job.CompletedTasks()) to await the status of the tasks.

Another approach is to use task dependencies, which require batchClient.Utilities.CreateTaskStateMonitor() and thus a statefull approach.

What is the best way to use Azure Batch in an event based design? And specifically, how to terminate/cleanup the jobs once the tasks are finished?

Solution

You can have the job "auto complete" once all tasks complete under the job. There is a property called OnAllTasksComplete on the CloudJob object.

You will want to initially set this property to NoAction (the default), while you are adding tasks to the job. After you have added all the tasks to the job, you can update that value to TerminateJob and then call Commit()/CommitAsync(). Note that if you retain the CloudJob that you initially submitted, you will need to Refresh()/RefreshAsync() first before modifying the properties and committing. Alternativley you can GetJob()/GetJobAsync(), modify, then commit.

For event-based designs, you can take a look at enabling Batch service analytics and see if that is appropriate for your scenario.