Search code examples
.netazurearchitecture

What is a good API architecture to handle long running tasks?

I'm trying to implement a ASP.NET Core based web service that's going to deploy Azure resources via ARM templates and the Azure SDK for .NET.

The thing is, based on my tests some operations can take up to three of four minutes to complete. I have no experience designing API's that take this long to do something.

So what I'm thinking of doing is the following.

  1. When API get's a request to provision a resource, I put the request information in a queue.
  2. I return a 200 with a ticket number for the caller to monitor the progress, which the caller can to by calling another endpoint.
  3. I use a webjob to create the resources request, and I update the information in the database.
  4. When the caller calls the endpoint to get feedback on the progress of the resource deployment I can now give the updated information (success or failure).

I'm not sure about this architecture, as I've never done nothing like this before. I'm thinking of using Azure Queues to organize the incoming requests, Azure Tables to put information regarding the requests as the deployments are made, and Azure WebJobs to perform the creation of the requests, as they are made.

Is this a decent enough architecture or should I consider using another technologies or patterns in order to do this? I'm not sure on how to handle this scenario and any input is welcome.

Solution

  • This question is soliciting opinions and is likely to be closed. But before that happens, this is a very reasonable plan for what you're trying to accomplish. I have personally implemented a very similar architecture that's held up nicely under heavy load. A few pointers:

    • Make sure you choose your queue implementation appropriately. Your choices are Service Bus Queues or Storage Queues. You can find a great comparison here.

    • Azure Tables are cheap (and can be lightning fast) storage that can REALLY scale. But you only have a partition key and a row key to work with. If this is sufficient for what you're trying to do, then great! Otherwise consider something like CosmosDB Tables or Azure SQL DB.

    • For long-running processing, take into consideration what happens if your webjob crashes and you have to restart processing a queue message. If there are multiple stages of processing that shouldn't be repeated, you need to plan for this. Otherwise the ServiceBusTrigger has nice retry logic built in.