K8s how to use more then 254 pods per node with flannel?

I'm trying to use 500+ pods per node in my cluster. I already set maxPods to 1k, and from node info I can see that this is successful.

However, flannel doesn't allow me to use more than 254 because it uses 24 subnet for pods, in range x.1 to x.255.

Is there a way to achieve this? I tried increasing pod-network-cidr to 8, and also configured flannel net-conf like so,

{                                                                                                                                                                   
  "Network": "10.0.0.0/8",                                                                                                                                         
  "SubnetLen": 16,                                                                                                                                                   
  "SubnetMin": "10.10.0.0",                                                                                                                                         
  "SubnetMax": "10.99.0.0",                                                                                                                                         
  "Backend": {                                                                                                                                                      
    "Type": "host-gw"                                                                                                                                               
  }                                                                                                                                                                 
}

But it doesnt seem to have an effect on my cluster.

Here is the event output of pending pods

rpc error: code = Unknown desc = failed to set up sandbox
 container "f5721f2419ad98fc8f26ad42f3f142a5733db38fefacc6d6f75cc38bd890233a" network for pod "test-deployment-f69647ffc-2bgb6": networkPlugin cni failed to set up
pod "test-deployment-f69647ffc-2bgb6_default" network: failed to allocate for range 0: no IP addresses available in range set: 10.0.0.1-10.0.0.254                 
  Warning  FailedCreatePodSandBox  8m18s (x80 over 30m)   kubelet            (combined from similar events): Failed to create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container "7045913eb55f3fbae10fe42de5fe29de73848e0cf005e96fa61286cdb127bc7d" network for pod "test-deployment-f69647ffc-2bgb6": net
workPlugin cni failed to set up pod "test-deployment-f69647ffc-2bgb6_default" network: failed to allocate for range 0: no IP addresses available in range set: 10.0.0.1-10.0.0.254

Solution

Solved it by changing node-cidr-mask-size of kube-controller-manager.

K8s:

Pass --node-cidr-mask-size 16 to kube-controller-manager and --pod-network-cidr=10.0.0.0/8 to kubeadm. edit kube-flannel.yml, net-conf.json array like the example below.

K3s:

export INSTALL_K3S_EXEC="--flannel-conf "<path-to-your-flannel-configuration>" --cluster-cidr "10.0.0.0/8" --kube-controller-manager-arg "node-cidr-mask-size=16"""

Edit your flannel configuration file like the example below, then install k3s as usual.

Example flannel configuration:

{
    "Network": "10.0.0.0/8",
    "Backend": {
        "Type": "vxlan"
    }
}