Azure functions to read some messages in queue

Question

consider this scenario: There is a queue which receives some messages:

1. New messages received
2. It starts processing them (the number of messages is not known but I need to know if the queue is empty), then I need to send an ok response to another Azure function or endpoint to do other tasks when all messages (s) been processed and the queue is empty.

The way the existing system works is that the queue is always monitored by calling an endpoint manually. I need to find a way to stop this continuous background task which has some overload on the system.

I am not sure which way is the best, having a Azure Service Bus trigger maybe instead of continuous monitoring? but not sure there is a way to find out that the queue is empty to send something to another Azure function.

I had some research I might be able to use https://learn.microsoft.com/en-us/azure/azure-functions/durable/durable-functions-overview?tabs=in-process%2Cnodejs-v3%2Cv1-model&pivots=csharp#monitoring . Any suggestions are welcome.

Answer 1

Make yourself a QueueClient object, perhaps in your Dependency Injection config in the Function app, and then check the ApproximateMessagesCount at the end of your Azure Function handler.

If there's less than a few messages in the count, then consider it empty. (don't forget to account for the 1 message that you're already processing)

public class MyQueueHandler
{
    private readonly QueueClient _queueClient;

    public MyQueueHandler(QueueClient queueClient)
    {
        _queueClient = queueClient; // From your DI config.
    }

    public async Task HandleQueueItemAsync([QueueTrigger("your-queue")] Something myQueueItem, FunctionContext context)
    {
        // TODO: Process your queue item as normal.
        await ProcessThisItemAsync(myQueueItem);

        // Check if the queue is nearly empty.
        QueueProperties properties = await _queueClient.GetPropertiesAsync();
        if (properties.ApproximateMessageCount <= 1)
        {
            // TODO: Whatever you need to notify about the queue being empty.
            await NotifyQueueNearlyEmpty();            
        }
    }
}

Caveat

Note you can never rely on the queue count being accurate. There will be many race conditions, as normal for a scalable Cloud architecture.

Even in a best case scenario, you could imagine that the queue looks empty when you check it, but then another message gets posted into the queue just a nanosecond later. You'd notify your other function that the queue is empty, when actually it might not be.

Likewise, I can't guarantee that the ApproximateMessagesCount will (or will not) include the message you're currently processing. Again I suspect there'll be a race condition.

So even with the above test for ApproximateMessageCount <= 1 there's no guarantee it will definitely pick up on an "empty" queue.

If you can give more info on the use case, and exactly how critical the count has to be, maybe people can give more guidance.

TL;DR: assume that the ApproximateMessagesCount will sometimes be wrong, and have something in-place to handle that, e.g.:

• A separate timer job that periodically checks the queue, and if it's somehow jumped to being empty then trigger your other process.
• Make sure your other process still runs correctly even if there are a few items left in the queue.
• Consider something with guaranteed message delivery (Service Bus etc) or other patterns, depending on use-case.

Reference: https://learn.microsoft.com/en-us/azure/storage/queues/queues-storage-monitoring-scenarios?tabs=azure-powershell#monitor-message-counts-in-each-queue