I am setting up the kubernetes cluster on CentOS 8 with containerd and Calico as CNI. with kubeadm command setup the master node, its in Ready status.
When I join the node to master, node not becoming ready status. I see below message the log file.
Jan 14 20:17:29 node02 containerd[1417]: time="2020-01-14T20:17:29.416373526-05:00" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:calico-node-fbst8,Uid:9c7f6334-d106-48e1-af12-1bcdebc7c2c2,Namespace:kube-system,Attempt:0,} failed, error" error="failed to start sandbox container: failed to create containerd task: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:279: applying cgroup configuration for process caused \"Invalid unit name 'pod9c7f6334-d106-48e1-af12-1bcdebc7c2c2'\"": unknown"
Jan 14 20:17:29 node02 kubelet[30113]: E0114 20:17:29.416668 30113 remote_runtime.go:105] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:279: applying cgroup configuration for process caused \"Invalid unit name 'pod9c7f6334-d106-48e1-af12-1bcdebc7c2c2'\"": unknown
Jan 14 20:17:29 node02 kubelet[30113]: E0114 20:17:29.416742 30113 kuberuntime_sandbox.go:68] CreatePodSandbox for pod "calico-node-fbst8_kube-system(9c7f6334-d106-48e1-af12-1bcdebc7c2c2)" failed: rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:279: applying cgroup configuration for process caused \"Invalid unit name 'pod9c7f6334-d106-48e1-af12-1bcdebc7c2c2'\"": unknown
Jan 14 20:17:29 node02 kubelet[30113]: E0114 20:17:29.416761 30113 kuberuntime_manager.go:729] createPodSandbox for pod "calico-node-fbst8_kube-system(9c7f6334-d106-48e1-af12-1bcdebc7c2c2)" failed: rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: OCI runtime create failed: container_linux.go:348: starting container process caused "process_linux.go:279: applying cgroup configuration for process caused \"Invalid unit name 'pod9c7f6334-d106-48e1-af12-1bcdebc7c2c2'\"": unknown
Jan 14 20:17:29 node02 kubelet[30113]: E0114 20:17:29.416819 30113 pod_workers.go:191] Error syncing pod 9c7f6334-d106-48e1-af12-1bcdebc7c2c2 ("calico-node-fbst8_kube-system(9c7f6334-d106-48e1-af12-1bcdebc7c2c2)"), skipping: failed to "CreatePodSandbox" for "calico-node-fbst8_kube-system(9c7f6334-d106-48e1-af12-1bcdebc7c2c2)" with CreatePodSandboxError: "CreatePodSandbox for pod \"calico-node-fbst8_kube-system(9c7f6334-d106-48e1-af12-1bcdebc7c2c2)\" failed: rpc error: code = Unknown desc = failed to start sandbox container: failed to create containerd task: OCI runtime create failed: container_linux.go:348: starting container process caused \"process_linux.go:279: applying cgroup configuration for process caused \\\"Invalid unit name 'pod9c7f6334-d106-48e1-af12-1bcdebc7c2c2'\\\"\": unknown"
Jan 14 20:17:30 node02 containerd[1417]: time="2020-01-14T20:17:30.541254039-05:00" level=error msg="Failed to load cni configuration" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Jan 14 20:17:30 node02 kubelet[30113]: E0114 20:17:30.541394 30113 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Jan 14 20:17:35 node02 containerd[1417]: time="2020-01-14T20:17:35.541792325-05:00" level=error msg="Failed to load cni configuration" error="cni config load failed: no network config found in /etc/cni/net.d: cni plugin not initialized: failed to load cni config"
Jan 14 20:17:35 node02 kubelet[30113]: E0114 20:17:35.541929 30113 kubelet.go:2183] Container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized
Any tips to resolve this error?
Because you are not using docker you need to setup the cgroup driver explicitly.
To use the systemd cgroup driver, set plugins.cri.systemd_cgroup = true in /etc/containerd/config.toml and systemctl restart containerd
You have to modify the file kubeadm-flags.env in /var/lib/kubelet and set the cgroups driver.
KUBELET_EXTRA_ARGS=--cgroup-driver=systemd
Make sure to point to above file in /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env