Search code examples
kubernetesnamespacesgoogle-kubernetes-enginekube-dns

Google GKE kubernetes DNS fail to resolve service name

I am having an issue with Kubernetes on GKE. I am unable to resolve services by their name using internal DNS.

this is my configuration

Google GKE v1.15

kubectl get namespaces
NAME              STATUS   AGE
custom-metrics    Active   183d
default           Active   245d
dev               Active   245d
kube-node-lease   Active   65d
kube-public       Active   245d
kube-system       Active   245d
stackdriver       Active   198d

I've deployed a couple of simple services based on openjdk 11 docker image and made with spring boot + actuator in order to have a /actuator/health endpoint to test in dev 

kubectl get pods --namespace=dev
NAME                          READY   STATUS    RESTARTS   AGE
test1-5d86946c49-h9t9l        1/1     Running   0          3h1m
test2-5bb5f4ff8d-7mzc8        1/1     Running   0          3h10m

If i try to execute under 

kubectl --namespace=dev exec -it test1-5d86946c49-h9t9  -- /bin/bash
root@test1-5d86946c49-h9t9:/app# cat /etc/resolv.conf
nameserver 10.40.0.10
search dev.svc.cluster.local svc.cluster.local cluster.local europe-west1-b.c.back-office-236415.internal c.back-office-236415.internal google.internal
options ndots:5
root@test1-5d86946c49-h9t9:/app# nslookup test2
Server:         10.40.0.10
Address:        10.40.0.10#53

** server can't find test2: NXDOMAIN

The same issue occurs if I try using test2 service and try to resolve test1. There is a special configuration for namespace to enable DNS resolve? Shouldn't this be automatic?

Solution

  • I have reproduced this using master version 1.15 and and type of service as ‘ClusterIP’. I am able to do look up from the Pod of one service to another. For creating Kubernetes Services in a Google Kubernetes Engine cluster [1] might be helpful.

    To see the services: $ kubectl get svc --namespace=default

    To access the deployment: $ kubectl exec -it [Pod Name] sh

    To lookup: $ nslookup [Service Name]

    Every Service defined in the cluster (including the DNS server itself) is assigned a DNS name. By default, a client Pod’s DNS search list will include the Pod’s own namespace and the cluster’s default domain.

    “Normal” (not headless) Services are assigned a DNS A record for a name of the form my-svc.my-namespace.svc.cluster-domain.example. This resolves to the cluster IP of the Service.

    For “Headless” (without a cluster IP) Services are also assigned a DNS A record for a name.Though this resolves to the set of IPs of the pods selected by the Service.

    However, DNS policies can be set on a per-pod basis. Currently Kubernetes supports the following pod-specific DNS policies. These policies are specified in the dnsPolicy field of a Pod Spec [2]:

    “Default“: The Pod inherits the name resolution configuration from the node that the pods run on.

    “ClusterFirst“: Any DNS query that does not match the configured cluster domain suffix, such as “www.kubernetes.io”, is forwarded to the upstream nameserver inherited from the node. Cluster administrators may have extra stub-domain and upstream DNS servers configured.

    “ClusterFirstWithHostNet“: For Pods running with hostNetwork, need to set its DNS policy “ClusterFirstWithHostNet”.

    “None“: It allows a Pod to ignore DNS settings from the Kubernetes environment. All DNS settings are supposed to be provided using the dnsConfig field in the Pod Spec.

    [1]-https://cloud.google.com/kubernetes-engine/docs/how-to/exposing-apps [2]-https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-config