Handling long-running tasks using Azure Functions and Azure Storage
I want to use Azure to handle long running tasks that can’t be handled solely by a web server as they exceed the 2 min HTTP limit (and would put unnecessary load on it regardless). In this case, it’s the generation of a PDF report that can take some time (between 2-5 mins). I’ve seen examples of solutions for this using other technologies (Celery, RabbitMQ, AWS Lamda, etc.) but not much using what's available on Azure (Functions and Storage in this case).
Details of my proposed solution are as follows (a diagram is here)
API (that has 3 endpoints):
- Generate report – post a message to Azure Queue Storage
- Get report generation status – query Azure Table Storage for status
- Get report – retrieve PDF from Azure Blob Storage
Azure Queue Storage
- Receives a message from the API containing parameters of the requested report
Azure Function
- Triggered when a message is added to Azure Queue Storage
- Creates report generation status record in Azure Table storage, set to ‘Generating’
- Generate a report based on parameters contained in the message
- Stores output PDF in Azure Blob Storage
- Updates report generation status record in Azure Table storage to ‘Completed’
Azure Table Storage
- Contains a table of report generation requests and associated status
Azure Blob Storage
- Stores PDF reports
Other points
- The app isn’t built yet – so there is no base case I’m comparing against (e.g. Celery/RabbitMQ)
- The time it takes to run the report isn’t super important (i.e. I’m not concerned about Azure Function cold starts)
- There’s no requirement for immediate notification of completion using something like Webhooks – the client will poll the API every so often using the get report generation endpoint.
- There won't be much usage of the app, so having an always active server to handle tasks (vs Azure Function) seems to be a waste of money.
- If I find that report generation takes longer than 10 mins, I can split it up into more than one Azure Function (to avoid consumption plan hard limit of 10 mins execution time per function)
My question is essential whether or not this is a good approach (to me it seems good, and relatively cost-effective, I’m just not sure if there’s something I’m missing).
This can be simplified using Durable Functions. Most of the job is already handled by the framework and you also can query an endpoint to check for the completion status.