Azure: how can I make sure Runbook invoke only one time per server

Question

I have virtual machines on Azure. Each master vm have few compute nodes running Slurm.

When a job finished, the compute nodes removed by Slurm shutdown script and the master stay running. I want to shutdown the master too.

I created an Azure Runbook that shutdown the master server. I can add line, to the script that shutdown the compute nodes, to invoke that Runbook.

The problem :

Each compute node will send request to shutdown his master, which is the same for few compute nodes. What cause to send the same request many times.

Is there any way to know in the Runbook that request to shutdown the specific mastrer was received and skip all the other request to shutdown the same master. Can I lock the Runbook or turn on a flag that the request was submitted?

Thanks.

Answer 1

Perhaps the easiest thing to do is to use Automation variables, like this:

$alreadySent = Get-AutomationVariable -Name ShutdownRequestSent
if (-not $alreadySent) {
    # Send shutdown request
    ...
    Set-AutomationVariable -Name ShutdownRequestSent -Value $true
}

You will have to figure out when and how to reset the variable to $false again.

This solution is simple but not entirely bullet-proof, and it may not be enough in your case. For example, if two jobs for this runbook reach the Get-AutomationVariable command around the same time, they will both send a shutdown request. Or, if the job sending a shutdown request, crashes for any reason before executing Set-AutomationVariable, the next job will send the second shutdown request. I don't know if this is a problem for you. If you need a stronger run-once guarantee, consider using Lease Blob.