Search code examples
dockerkuberneteskubeadmcoredns

Kubernetes V1.11.0 HA cluster CoreDNS PODS not coming up


I am trying to create Kubernetes V1.11.0 HA cluster in Centos 7 server. I have IPV6 disabled on my host.

https://kubernetes.io/docs/setup/independent/high-availability/

Based on these steps added all three master, coreDNS POD is not coming up, its keep logging below error in /var/log/messages file.

kubectl get pods -n kube-system
NAME                                READY     STATUS              RESTARTS   AGE
coredns-78fcdf6894-8t7bt            0/1       ContainerCreating   0          53m
coredns-78fcdf6894-plmll            0/1       ContainerCreating   0          53m

error message:

Jul 17 10:09:28 master03 kubelet: E0717 10:09:28.531244   10041 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = [failed to set up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to set up pod "coredns-78fcdf6894-8t7bt_kube-system" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to teardown pod "coredns-78fcdf6894-8t7bt_kube-system" network: failed to get IP addresses for "eth0": <nil>]
Jul 17 10:09:28 master03 kubelet: E0717 10:09:28.531286   10041 kuberuntime_sandbox.go:56] CreatePodSandbox for pod "coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)" failed: rpc error: code = Unknown desc = [failed to set up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to set up pod "coredns-78fcdf6894-8t7bt_kube-system" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to teardown pod "coredns-78fcdf6894-8t7bt_kube-system" network: failed to get IP addresses for "eth0": <nil>]
Jul 17 10:09:28 master03 kubelet: E0717 10:09:28.531298   10041 kuberuntime_manager.go:646] createPodSandbox for pod "coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)" failed: rpc error: code = Unknown desc = [failed to set up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to set up pod "coredns-78fcdf6894-8t7bt_kube-system" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container "8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c" network for pod "coredns-78fcdf6894-8t7bt": NetworkPlugin cni failed to teardown pod "coredns-78fcdf6894-8t7bt_kube-system" network: failed to get IP addresses for "eth0": <nil>]
Jul 17 10:09:28 master03 kubelet: E0717 10:09:28.531358   10041 pod_workers.go:186] Error syncing pod 7caa3f29-89c3-11e8-aa0f-00505693ca30 ("coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)"), skipping: failed to "CreatePodSandbox" for "coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)" with CreatePodSandboxError: "CreatePodSandbox for pod \"coredns-78fcdf6894-8t7bt_kube-system(7caa3f29-89c3-11e8-aa0f-00505693ca30)\" failed: rpc error: code = Unknown desc = [failed to set up sandbox container \"8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c\" network for pod \"coredns-78fcdf6894-8t7bt\": NetworkPlugin cni failed to set up pod \"coredns-78fcdf6894-8t7bt_kube-system\" network: open /proc/sys/net/ipv6/conf/eth0/accept_dad: no such file or directory, failed to clean up sandbox container \"8cf72126bd4ea7e33806512997d02b51c3f95331701928c52c19d1ddea9d6f9c\" network for pod \"coredns-78fcdf6894-8t7bt\": NetworkPlugin cni failed to teardown pod \"coredns-78fcdf6894-8t7bt_kube-system\" network: failed to get IP addresses for \"eth0\": <nil>]"
Jul 17 10:09:28 master03 kubelet: W0717 10:09:28.636173   10041 cni.go:243] CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "88183d8a24dcafca5b04e3a6454e83aca1417a6934f50cd52b3e6fffbae4f272"
Jul 17 10:09:28 master03 kernel: XFS (dm-18): Mounting V4 Filesystem

How to fix this error message and bring the coreDNS PODs?

Thanks


Solution

  • I have to enable the IPv6 to my CoreDNS pod to run.

    Here is the steps I followed:

    https://www.thegeekdiary.com/how-to-enable-ipv6-on-centos-rhel-7/

    Edit /etc/default/grub and change the value of kernel parameter ipv6.disable from 1 to 0 in line:

    # grub2-mkconfig -o /boot/grub2/grub.cfg
    
    # shutdown -r now