I'm training a Neural Network on an Azure Spot Instance VM and I want to be able to run a shutdown script that saves my model when the spot instance is evicted. Google Cloud has an option for a shutdown script. However, I haven't found an equivalent feature in Microsoft Azure. Essentially, I need to be able to send a SIGKILL signal to my program when my instance is being evicted. Does anyone have an idea of how to do this?
The closest I can think of is Azure Instance Metadata service which can be used for your scenario.
Instance Metadata service essentially provides an endpoint which you should invoke from inside your spot instance frequently. You can get the below events from the endpoint.
Thus, the endpoint will let you know when the VM is going to be evicted. as per the links below, it gives a 30 seconds headsup wherein you may be able to do you clean up activities.
You can find more details here [https://learn.microsoft.com/en-us/azure/virtual-machines/windows/instance-metadata-service]
A good article on how to use the IMS can be found here.
Hope that helps