I have two Azure Queue Triggers that run sequentially, one a WebJob and the other an Azure Function. The first queue trigger generates a number of messages that get picked up by the second queue trigger. The problem is that the first queue trigger generates a blob that the second queue triggers all use, and I want to delete that blob after the second queue triggers all finish.
The most obvious solution is to convert the queues to durable functions with the fan-out/fan-in design pattern, but I would prefer to avoid this as the queues give me automatic retry/poison queues and are already set up and functioning with other existing code that does not need to delete the blob after the second queues all finish.
I've come up with a couple potential solutions but wanted feedback as I'm still relatively new to Azure Functions and WebJobs. They both essentially use workarounds to asynchronously count the completion of the second queue triggers and delete the blob from the second queue trigger when the last one has completed.
Solution 1:
Have the first queue trigger create a temporary queue. Before adding the original queue messages to the second queue, add an equal number of queue messages as those original queue messages to the temporary queue. After each time the second queue trigger finishes processing one of the original messages, dequeue a message from the temporary queue. Delete the blob and temporary queue after last item is dequeued from the temporary queue.
Solution 2:
Create a table in the storage account. Before the queue messages are added to the second queue, insert a row with a blob name and total message count to table. After the second queue trigger finishes processing one of the messages, insert a row with the blob name. After the number of inserted rows equals the count, delete the blob and table rows.
There are other potential solutions I've come up with such as using a Redis Cache instead of a table in the storage account or removing the temporary file after a period of time instead of trying to count the number of completions, but the two solutions above seem the most promising so far.
Does anyone with more experience using Azure Functions and WebJobs have any suggestions?
Solution 2 is very similar to what Durable Functions do behind the scenes, So, it will work, but the burden of implementation and debugging is on you. You might get into some concurrency issues, uncertainties while handling retries etc. If your solution doesn't need to be very robust, go for it.
If the problem is just deleting blob files, I can imagine a simpler solution. E.g. sending another message deferred by e.g. 1 hour to delete it, while you know that your processing time is always less than one hour. Or just have a periodic job to clean up old files.
I doubt that an alternative and trivial and correct solution exists for fan-in.