I am facing the following problem: I need to execute on-demand long running workers on Azure VMs. These workers are wrapped in a docker image.
So I looked at what Azure is offering and I seem to have the following two options:
Use a VM with docker-compose. This means I need to be able to programatically start a VM, run the docker image on it, and then shutdown the VM (the specs we use are quite expensive and we can't let it run indefinitely). However this means writing orchestration logic ourselves. Is there a service that maybe we could use to make life easier?
Setting up a k8s cluster. However, I am not sure how pricing works here. Would I be able to add the type of the VMs we use to the cluster, and then use the k8s API to start on-demand containers? How would I get priced in this case?
If the only thing you need are workers, there are a few more options you have. Which service suits best depends on the requirements you have. Based on what's in your question, I would think one of the following two might fit best:
Azure Container Instances offers the fastest and simplest way to run a container in Azure, without having to manage any virtual machines and without having to adopt a higher-level service.
Azure Container Instances is a great solution for any scenario that can operate in isolated containers, including simple applications, task automation, and build jobs.
Azure Container Apps enables you to run microservices and containerized applications on a serverless platform. Common uses of Azure Container Apps include:
- Deploying API endpoints
- Hosting background processing applications
- Handling event-driven processing
- Running microservices
According to Azure's Container services page, here are your options:
| IF YOU WANT TO | USE THIS |
|---|---|
| Deploy and scale containers on managed Kubernetes | Azure Kubernetes Service (AKS) |
| Deploy and scale containers on managed Red Hat OpenShift | Azure Red Hat OpenShift |
| Build and deploy modern apps and microservices using serverless containers | Azure Container Apps |
| Execute event-driven, serverless code with an end-to-end development experience | Azure Functions |
| Run containerized web apps on Windows and Linux | Web App for Containers |
| Launch containers with hypervisor isolation | Azure Container Instances |
| Deploy and operate always-on, scalable, distributed apps | Azure Service Fabric |
| Build, store, secure, and replicate container images and artifacts Azure | Container Registry |
EDIT:
Based on the comment
Let's say the only requirement is that I am able to use the resources on-demand, so I only end up spending the amount of money that would take for a certain job to finish execution. What would you use?
the answer would most probably be Container Apps, if the code you have available is not easily migrated to an Azure Function. The most important reason: they are Serverless, which means they scale to zero and you only pay for actual consumption. Next to that, you have to write limited to no orchestration logic, since the container apps can scale based on event sources.
Enables event-driven application architectures by supporting scale based on traffic and pulling from event sources like queues, including scale to zero.
Another great resource is Comparing Container Apps with other Azure container options.