Search code examples
azure-functionsazure-monitoringazure-durable-functionsazure-alerts

How to Setup Azure Alert for Failed Durable Function Orchestration?

I would like to be notified via an email alert when an unknown exception occurs in an activity function that is part of a durable orchestration chain. The problem is that when an exception is thrown in an Activity Function, the logged error in Application Insights shows a status code of zero. In Azure Alerts, there is no alert configuration that will look for a log message status code of zero, only 1xx, 2xx, 4xx, etc. No matter what I've tried on code, I can't get the functions to return a 4xx of 5xx error code.

How can I setup in Azure Monitor Alert to catch a failed orchestration? Is there a way to force a non-zero status code when an exception is thrown?

Background Info

As the following screenshot shows, Azure Monitor for the durable function is showing a failure, which is what I'm expecting...

enter image description here

However, I can't figure-out how to setup an alert to detect this failure. Finding the exception in Application Insights, the response code that caused the failure is zero...

enter image description here

Going over to Azure Monitor Alerts, when setting-up a new alert to detect this condition, there is no such thing as an alert condition for a status code of zero (which makes sense, but doesn't help me)...

enter image description here

Here is a shell of the Activity Function that is intentionally throwing an exception to help me setup the alert and know it's working...

[FunctionName("SearchProductIndexFunctionActivity")]
public static async Task Run(
    [ActivityTrigger]string input,
    ILogger logger,
    ExecutionContext context)
{
    logger.LogInformation("Started SearchProductIndexFunctionActivity...");

    throw new ArgumentException("Oops...");

    logger.LogInformation("Finished SearchProductIndexFunctionActivity successfully.");
}

And here is the Orchestrator Function that calls the Activity Function. If all exceptions are not caught and rethrown, the Orchestrator Function shows a "success" status which isn't helpful... I want to know something failed. However, the response status code of zero that results from this exception doesn't help setup the alert. Any suggestions on how to fix this?

[FunctionName("SearchOrchestration")]
public static async Task Run(
    [OrchestrationTrigger]DurableOrchestrationContext orchestrationContext,
    ExecutionContext context,
    ILogger logger)
{
    try
    {
        await orchestrationContext.CallActivityAsync(PART_INDEX_UPDATE_ACTIVITY_NAME, null);

        await orchestrationContext.CallActivityAsync(PRODUCT_INDEX_UPDATE_ACTIVITY_NAME, null);
    }
    catch (Exception ex)
    {
        logger.LogError(ex, $"Search index orchestration failed.");
        throw;
    }
}
Solution

  • Since you're using Application insights, you can consider using create a custom log search alert as per this doc.

    For "Resource", you should select the application insights used for the azure function.

    For "Condition", select the "Custom log search". Then in the "Search query" textbox, you can write a query. In your case, the error should in the exception table of application insights, so the query looks like exceptions | your_query_for_error.

    If you are not familiar with the sql-like query in application insights, you can refer to this article. And you can write your query in application insights first to make sure it can work as expected. A screenshot of query in application insights:

    enter image description here