Kubernetes 1.24+ fails with flannel giving "cni plugin not initialized"

Question

I'm doing a single node kubernetes installation (so control-plane (or master pre-1.24) taint removed) on Ubuntu 20.04. For versions of kubernetes after 1.23 the network wont initialize. Everything works on versions prior to 1.24. Is there something obvious I'm doing wrong?

My init command is:

sudo kubeadm init --apiserver-advertise-address=$IP_TO_BIND_TO --pod-network-cidr=10.244.0.0/16

I apply flannel using:

kubectl apply -f https://raw.githubusercontent.com/flannel-io/flannel/master/Documentation/kube-flannel.yml

The node remains "NotReady" and the describe nodes gives me:

...
  Ready                False   Wed, 21 Sep 2022 08:27:47 -0400   Wed, 21 Sep 2022 08:27:32 -0400   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
...

The file /etc/cni/net.d/10-flannel.conflist is there and contains:

{
  "name": "cbr0",
  "cniVersion": "0.3.1",
  "plugins": [
    {
      "type": "flannel",
      "delegate": {
        "hairpinMode": true,
        "isDefaultGateway": true
      }
    },
    {
      "type": "portmap",
      "capabilities": {
        "portMappings": true
      }
    }
  ]
}

Flannel is running; kubectl get pods --all-namespaces gives:

AMESPACE      NAME                                  READY   STATUS    RESTARTS   AGE
kube-flannel   kube-flannel-ds-22w4k                 1/1     Running   0          42s
kube-system    coredns-565d847f94-fl2b6              0/1     Pending   0          42s
kube-system    coredns-565d847f94-qnzs4              0/1     Pending   0          42s
kube-system    etcd-master-node                      1/1     Running   0          56s
kube-system    kube-apiserver-master-node            1/1     Running   0          56s
kube-system    kube-controller-manager-master-node   1/1     Running   0          56s
kube-system    kube-proxy-mnwcm                      1/1     Running   0          42s
kube-system    kube-scheduler-master-node            1/1     Running   0          56s

The coredns pods are awaiting the NotReady taint which I'm assuming can't be removed without fixing the network problem cni plugin not initialized

The file /run/flannel/subnet.env is also there and contains:

FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

ifconfig shows a cni0 interface:

cni0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 10.244.0.1  netmask 255.255.255.0  broadcast 10.244.0.255
        inet6 fe80::b852:baff:fe10:4f4b  prefixlen 64  scopeid 0x20<link>
        ether ba:52:ba:10:4f:4b  txqueuelen 1000  (Ethernet)
        RX packets 286  bytes 25159 (25.1 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 357  bytes 65069 (65.0 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

apparmor and ufw are disabled.

Any suggestions on how to get this running would be appreciated.

EDIT: It turns out that if I have the wifi disabled in the Ubuntu NetworkManager, even though the machine is wired and the $IP_TO_BIND_TO in the kubeadm init call (see above) is the wired enabled network interface, as of 1.24, the cni plugin fails to initialize.

Answer 1

The problem appears to be that I'm applying the cni network plugin (flannel or calico) too soon after starting kubelet for the first time; all of this is scripted in my system.

If you wait until the coredns pods are in a Pending state and THEN kubectl apply the network, whether calico or flannel, it seems to work.

This wasn't the case prior to 1.24.