Search code examples
azureazure-resource-managerazure-akscidrvnet

Azure AKS deployment raised an "InsufficientSubnetSize" error


I'm trying to deploy an Azure AKS instance via ARM template.
I have a requirement to integrate the AKS instance into an existing Vnet.
I have a dedicated subnet for AKS service.
However, deployment has failed with the following error:

{"code":"DeploymentFailed","message":"At least one resource deployment operation  failed.  
Please list deployment operations for details. Please see  
https://aka.ms/arm-debug for usage details.","details":  
[{"code":"BadRequest","message":"{\r\n \"code\": \"InsufficientSubnetSize\",\r\n  
\"message\": \"Pre-allocated IPs 93 exceeds IPs available in Subnet 11\",\r\n  
\"target\": \"agentPoolProfile.count\"\r\n}"}]}  

I'm using the following address space for Vnet: XX.XX.XX.0/24 (XX.XX.XX.0 - XX.XX.XX.255 which has 256 addresses.
I have a set of dedicated subnets within this Vnet, each of /28 mask (11+5 addresses depth):

XX.XX.XX.0/28  
XX.XX.XX.16/28  
XX.XX.XX.64/28  
XX.XX.XX.128/28  
XX.XX.XX.144/28  
XX.XX.XX.160/28  
XX.XX.XX.176/28 

The subnet XX.XX.XX.144/28 is planned to be used in AKS.
The current AKS instance ARM template is as follows:

"resources": [
        {
            "type": "Microsoft.ContainerService/managedClusters",
            "apiVersion": "2019-04-01",
            "name": "[parameters('resourceName')]",
            "location": "[parameters('location')]",
            "dependsOn": [],
            "tags": {},
            "properties": {
                "kubernetesVersion": "[parameters('kubernetesVersion')]",
                "enableRBAC": "[parameters('enableRBAC')]",
                "dnsPrefix": "[parameters('dnsPrefix')]",
                "agentPoolProfiles": [
                    {
                        "name": "agentpool",
                        "osDiskSizeGB": "[parameters('osDiskSizeGB')]",
                        "count": "3",
                        "vmSize": "[parameters('agentVMSize')]",
                        "osType": "[parameters('osType')]",
                        "storageProfile": "ManagedDisks",
                        "maxPods": "30",
                        "vnetSubnetID": "/subscriptions/XXXXX/resourceGroups/XXXX/providers/Microsoft.Network/virtualNetworks/VNET_NAME/subnets/akssubnet"
                    }
                ],
                "servicePrincipalProfile": {
                    "ClientId": "[parameters('servicePrincipalClientId')]",
                    "Secret": "[parameters('servicePrincipalClientSecret')]"
                },
                "networkProfile": {
                    "networkPlugin": "azure",
                    "serviceCidr": "10.0.0.0/16",
                    "dnsServiceIP": "10.0.0.10",
                    "dockerBridgeCidr": "172.17.0.1/16"
                },
                "addonProfiles": {
                    "httpApplicationRouting": {
                        "enabled": "[parameters('enableHttpApplicationRouting')]"
                    },
                    "omsagent": {
                        "enabled": "[parameters('enableOmsAgent')]",
                        "config": {
                            "logAnalyticsWorkspaceResourceID": "[parameters('omsWorkspaceId')]"
                        }
                    }
                }
            }
        },        
            "subscriptionId": "[split(parameters('omsWorkspaceId'),'/')[2]]",
            "resourceGroup": "[split(parameters('omsWorkspaceId'),'/')[4]]"
        }
    ]

Network profile parameters were set according to the following article: Microsoft.ContainerService managedClusters template reference

A CIDR of 10.0.0.0/16 is of a private range and isn't interfering with my existing Vnet range.

I need advice on how to deal with this deployment error.

Upd:
I've tried the deployment with the values of my Vnet/subnets but stil it's failing:
enter image description here

Upd2:

Per MS documentation "Minimum number of pods on the initial cluster creation using Azure CNI type is 30" which leads to the following number of subnet range in my case according to the formula: (number of nodes + 1) + ((number of nodes + 1) * maximum pods per node that you configure) = (3+1) + ((3+1)*30) = 124

So the multiplier of 30 will be always present even if the number of pods is set to 1 in ARM template for example.

Upd3:

However, as I was unable to extend the existing subnet range I've managed to deploy the AKS instance using the following configuration:

"parameters": {
 "SvcCidr": {
      "type": "string",
      "defaultValue": "10.0.0.0/16",
      "metadata": {
        "description": "Maximum number of pods that can run on a node."
      }
    },
    "PodCidr": {
      "type": "string",
      "defaultValue": "10.244.0.0/16",
      "metadata": {
        "description": "Maximum number of pods that can run on a node."
      }
    },
    "DnsSvcIP": {
      "type": "string",
      "defaultValue": "10.0.0.10",
      "metadata": {
        "description": "Maximum number of pods that can run on a node."
      }
    },
    "DockerCidr": {
      "type": "string",
      "defaultValue": "",

"variables": {
    "vnetSubnetId": "[resourceId(resourceGroup().name, 'Microsoft.Network/virtualNetworks/subnets', parameters('vnetName'), parameters('vnetSubnetName'))]",

"resources": [
{
      "type": "Microsoft.ContainerService/managedClusters",
 "agentPoolProfiles": [
          {
      "vnetSubnetID": "[variables('vnetSubnetId')]",
 "networkProfile": {
          "networkPlugin": "[parameters('NetPlugin')]",
          "serviceCidr": "[parameters('SvcCidr')]",
          "podCidr": "[parameters('PodCidr')]",
          "DNSServiceIP": "[parameters('DnsSvcIP')]",
          "dockerBridgeCidr": "[parameters('DockerCidr')]"

Which leads to the provision of my subnet range IP addresses only to cluster nodes while the pods will use the private IP addresses range.


Solution

  • For your issue, when you use the azure module network, as it shows about the Calculation method in other answers, your subnet could just have one node. But actually, the IP address number of your subnet is not enough for just one node. Because there are already pods need the IP address when you create the AKS cluster in default, for example, the metric server and etc.

    So you just can use the network nodule kubelet. In this module, just the node need to IP address in the subnet. And just use this network module, you can have 3 nodes as you want and use your existing subnet with just 8 IP address. For more details, see Use kubenet networking with your own IP address ranges in Azure Kubernetes Service (AKS).