Search code examples
azurenotificationsinfrastructureazure-cloud-services

Azure Role Failure Notifications

I have an Azure Cloud Service Load Balanced over 2 Roles with a recent addition of a custom LoadBalancerProbe.

The issue I am having is that in my site I have a threaded task that was throwing unhandled exceptions, 5 of these unhandled exceptions was causing the role to report itself as unhealthy. I have fixed this issue by dealing with these unhandled exceptions, however, we have a large team and I am fearful that this may happen again as a result of a future release.

I would like to set up an azure notification (preferably email) to let me know when one of my roles reports itself unhealthy. I have looked around on the web but cannot find any help. Has someone done this before? It's more of a precaution but I am keen to get this implemented.

Thanks in advance for any help you can give me

Solution

  • There isn't an alert baked into the platform for this that I know of, but you can certainly do some coding/scripting to get this done. You can use any of the SDKs, Microsoft Azure Management Libraries or command line tools that let you get at the Azure Management API, but for my example here I'll use PowerShell. I don't have a script to do this completely, but the main part is getting the instance statuses, such as:

    $NonReadyInstances = (Get-AzureDeployment $serviceName -Slot Production -verbose:$false).RoleInstanceList | Where-Object { $_.InstanceStatus -ne "ReadyRole" }

    This is getting all the instances that aren't in a ready state; however, you'd probably want to filter out for several of the other valid states of instances as your system scales, etc. There is a list of valid instances states you can use to decide which ones you want to be notified about. For example, in your case I'd think CyclingRole or RestartingRole would be ones to watch for.

    Check the count of how many items are returned in the $NonReadyInstances variable and if it is greater than zero send your email. Put that in a script and run it regularly.

    Here's a link to getting started and set up with the Azure PowerShell Cmdlets if you aren't familiar with it. - http://msdn.microsoft.com/en-us/library/jj156055.aspx

    Also, here is a link on the Azure Automation preview service if you want to run this without running it from your own machines- http://azure.microsoft.com/en-us/services/automation/.