Search code examples
azureazure-aksazure-container-instances

Is it possible to use multiple virtual nodes on an AKS cluster

I have an AKS cluster with two nodes. One is a classic VM, and the other one is a Virtual Node. Since I have different subscriptions across my account, I want to deploy a Virtual Node to each of them. So I can track the billing of the usage for every subscription. Is there a way to create a Virtual Node across multiple subscriptions and add them to the AKS cluster?

I have created an AKS cluster with the Virtual Nodes feature enabled. I can see the virtual node, and I can deploy pods on it.

Solution

  • Is there a way to create a Virtual Node across multiple
    subscriptions and add them to the AKS cluster? So, I can track the billing of the usage for every subscription.

    Azure Kubernetes supports Multi Tenancy deployment where AKS resources can be shared or deployed separately across multiple tenants residing in different subscriptions.

    In order to automate the deployment across multiple subscriptions you can make use of IAC infrastructure as a code tools like ARM templates, Bicep, Terraform etc.

    Approachs:

    1) Vertically partitioned Deployments : -

    enter image description here

    Here, Tenant A and Tenant B run on same Node pools connected to same AKS cluster and C runs on isolated node pool. Consider these node pools as your virtual nodes backed by Azure Container Instance with virtual kubelet which auto-scales the nodes whenever AKS needs it.
    By autoscaling the Virtual nodes, you will have multiple nodes running in your AKS cluster.

    2) Horizontal partitioned Deployments :-

    enter image description here

    In this method you can have a shared AKS cluster across multiple Azure subscriptions and Different Nodes running individually in every subscription or tenant, As in the above diagram Databases are deployed individually across multiple subscriptions.

    3) Automated Single Tenant Deployments :-

    enter image description here

    In this deployment model, you deploy dedicated resources in each subscriptions via a automated code with either bicep, ARM, or terraform via a pipeline. you can refer the above image from this Document.

    You can deploy this multi-tenant architecture with the use of Deployment stamps to automate the deployment of virtual nodes backed by ACI across multiple subscriptions.

    Deployment stamp pattern helps in deploying multiple Azure resources across multiple tenants and subscriptions. You assign a stamp to multiple resources and deploy those resources in your tenant. Then the copy of the same stamp is deployed in another tenants.

    You can enable the Azure virtual Node add on to your AKS by adding the command to add the node to your cluster in the ARM template, IAC deployment in pipeline itself.
    You can create either a Bicep or Terraform or ARM Template to deploy your Virtual nodes across multiple subscriptions by running them via a pipeline with git-hub actions or via Azure Devops. You can create different stages or builds for different subscriptions and initiate a parallel build to automate the deployment in Azure DevOps.

    You can refer this git-hub repository to deploy a virtual node auto-scaler in your tenant :-

    Azure-Samples/virtual-node-autoscale: A sample application to demonstrate Autoscale with AKS Virtual Nodes (github.com)

    You can create your own ARM template or github repo with the code files and run it via Azure Devops pipeline with examples: -

    ARM Template to deploy AKS: -

    enter image description here

    Reference: -

    Walkthrough: Create Azure Kubernetes Service (AKS) using ARM template – AzureDays

    ARM Template with parameter Stamp:-

    enter image description here

    Reference:

    Deployment Stamps pattern - Azure Architecture Center | Microsoft Learn

    You can make use of Nested ARM template and use multiple resource in a parent child combination to deploy multiple resources with single template.

    You can automate the deployment across multiple subscriptions by using Azure Devops or via Github actions:-

    Azure DevOps pipeline -
    enter image description here

    And to track the billing data you can create an Alert in every subscription to get an email of the Usage consumption of the nodes. Refer here -Azure cost analysis.

    Reference : - Deploy multiple instances of resources - Azure Resource Manager | Microsoft Learn.