Search code examples
kuberneteskubeadmkube-dnsweave

kube-dns with weave failing to lookup service

I created a kubeadm (Kubernetes 1.8) cluster on my Fedora machine with one vagrant node. The cluster is running fine but I am facing a weird issue when I test my dns:

$ kubectl exec busybox -- nslookup friendservice.mynamespace
Server:    10.96.0.10
Address 1: 10.96.0.10 kube-dns.kube-system.svc.cluster.local

Name:      friendservice.mynamespace
Address 1: 10.44.0.2 friendservice-
0.friendservice.mynamespace.svc.cluster.local

$ kubectl -n mynamespace exec userservice-0 -- nslookup 
friendservice.mynamespace
nslookup: can't resolve '(null)': Name does not resolve

Name:      friendservice
Address 1: 10.44.0.2 friendservice-
0.friendservice.mynamespace.svc.cluster.local

nslookup from a busybox pod in the default namespace of a service running in the mynamespace namespace is working fine, but it seems when I try to do nslookup of a service in the same custom namespace (mynamespace) then dns first fails to resolve but then resolves. What am I missing here?

$ kubectl get pods --all-namespaces
kubectl get pods --all-namespaces         
NAMESPACE     NAME                                    READY     STATUS             RESTARTS   AGE
default       busybox                                 1/1       Running            2          2h
kube-system   etcd-fed-master                         1/1       Running            6          2h
kube-system   kube-apiserver-fed-master               1/1       Running            0          2h
kube-system   kube-controller-manager-fed-master      1/1       Running            0          2h
kube-system   kube-dns-545bc4bfd4-jkhrr               3/3       Running            0          2h
kube-system   kube-proxy-5vcvr                        1/1       Running            0          2h
kube-system   kube-proxy-f4765                        1/1       Running            0          2h
kube-system   kube-scheduler-fed-master               1/1       Running            1          2h
kube-system   weave-net-jw647                         2/2       Running            0          2h
kube-system   weave-net-z25rv                         2/2       Running            0          2h
mynamespace     friendservice-0                        1/1       Running            5          10m
mynamespace     userservice-0                          1/1       Running   0          26m

$ kubectl exec busybox -- cat /etc/resolv.conf 
nameserver 10.96.0.10
search default.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

$ kubectl -n mynamespace exec bookentryservice-0 -- cat /etc/resolv.conf
nameserver 10.96.0.10
search mynamespace.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Any help will be greatly appreciated.

Solution

  • This is a problem with Alpine Linux and its musl library. It has broken DNS functionality and it has been this way for years and they apparently aren't really bothered to fix it.

    https://github.com/gliderlabs/docker-alpine/blob/master/docs/caveats.md#dns https://github.com/gliderlabs/docker-alpine/issues/8