Search code examples
kuberneteskubernetes-podcoredns

coredns crashes with error "Failed to list *v1.Service: Get https://10.96.0.1:443/api/v1/****: dial tcp 10.96.0.1:443: connect: no route to host"


CoreDNS pod is not running. Please find below status.

kubectl get po --all-namespaces -o wide | grep -i coredns
kube-system            coredns-6955765f44-8qhkr                    1/1     Running            0          24m     10.244.0.59      k8s-master          <none>           <none>
kube-system            coredns-6955765f44-lpmjk                    0/1     Running            0          24m     10.244.1.43      k8s-worker-node-1   <none>           <none>

Please find below logs of pod.

kubectl logs coredns-6955765f44-lpmjk -n kube-system



E0420 03:43:03.855622       1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host
E0420 03:43:03.855622       1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host
E0420 03:43:03.855622       1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host
E0420 03:43:03.855622       1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host
E0420 03:43:05.859525       1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host
E0420 03:43:05.859525       1 reflector.go:125] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:98: Failed to list *v1.Namespace: Get https://10.96.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 10.96.0.1:443: connect: no route to host

Solution

  • To solve no route to host issue with CoreDNS pods you have to flush iptables by running:

    systemctl stop kubelet
    systemctl stop docker
    iptables --flush
    iptables -tnat --flush
    systemctl start kubelet
    systemctl start docker
    

    Also mind that flannel has been removed from the list of CNIs in the kubeadm documentation:

    The reason for that is that Cluster Lifecycle have been getting a number of issues related to flannel (either in kubeadm or kops tickets) and we don't have good answers for the users as the project is not actively maintained. - Add note that issues for CNI should be logged in the respective issue trackers and that Calico is the only CNI we e2e test kubeadm against.

    So recommended approach would be also move to Calico CNI.