Azure Durable Orchestration timeout WaitForExternalEvent loses input because of polling times

Question

We have an Azure Orchestration that waits for an external event or a timeout. The timeout is 30 seconds. The problem is that the handling of the timeout happens around 20 seconds late consistently. This happens because of the polling interval.

Our issue now is that, if the input AlarmCommand happens between the timeout expiring and the actual function running (~20sec later), the input is lost.

This is an implementation from an official Microsoft doc example found here.

TimeSpan timeout = // 30 seconds
DateTime deadline = context.CurrentUtcDateTime.Add(timeout);

using var cts = new CancellationTokenSource();
{
    var alarmTask = context.WaitForExternalEvent<AlarmMessage>("AlarmCommand");
    var timeoutTask = context.CreateTimer(deadline, cts.Token);
    _logger?.LogWarning($"Deadline: {deadline}");
    var winner = await Task.WhenAny(alarmTask, timeoutTask);

    if (winner == alarmTask)
    {
        cts.Cancel();
        _logger?.LogWarning($"Input detected");
        // success
    }
    else
    {
        _logger?.LogWarning($"Timeout reached");
        // timeout
    }
};

The function is running late sometimes because of the way Azure polls Orchestrations. The timeout isn't triggered until the next polling happens. The problem is that any AlarmCommand input given between the timeout expiration and when the Orchestration actually triggers is lost.

How can we prevent AlarmCommand from being lost in this case?

Thanks!

Answer 1

Durable Functions in Azure have a default maxQueuePollingInterval of 30 seconds. This means that the function won't notice any timers expiring until up to 30 seconds after the expiration time.

You could change the interval to 1 second. That way, the expired timeout will be handled up to 1 second instead of up to 30 seconds late.

Add the following to host.json

"extensions": {
    "durableTask": {
        "storageProvider": {
            "maxQueuePollingInterval": "00:00:01"
        }
    }
}