Pods don't seem to be receiving upstream dns or stub domains when using custom kube-dns config map

I have a setup a config map called kube-dns in the kube-system NS, with some custom upstream DNS entries (3 x of them), but my containers/pods are not inheriting these custom DNS entries in their resolv.conf files when I schedule them through a simple deployment.

My logs in the kube-dns pods don't seem to be pointing at the name of the config map (it looks like an empty string). Could this be the problem?

After adding the custom config map, I did delete the kube-dns pods, and allowed the existing kube-dns deployment to re-create the pods (there are 2 x sets of kube-dns pods that were terminated, and re-created).

I used this guide to set up my config map (I a blog post around the feature that was introduced with 1.6):

https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/#configmap-options

http://blog.kubernetes.io/2017/04/configuring-private-dns-zones-upstream-nameservers-kubernetes.html

Here is my config map:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
data:
  stubDomains: |
    {"myinternaldomainhere.net": ["10.254.131.155"]}
  upstreamNameservers: |
    ["10.254.131.155", "8.8.8.8", "8.8.4.4"]

Is there somewhere else that I need to specify that the deployment of kube-dns references the name of the config map? In the pod logs, I can see the flag for the config map names seems to be an empty string.

Logs for the new kubedns pods that I looked at after deleting the old pods say (notice line 5 is where I see the empty string reference):

I1110 16:35:35.685518       1 dns.go:48] version: 1.14.4-2-g5584e04
I1110 16:35:35.686074       1 server.go:70] Using configuration read from directory: /kube-dns-config with period 10s
I1110 16:35:35.686136       1 server.go:113] FLAG: --alsologtostderr="false"
I1110 16:35:35.686148       1 server.go:113] FLAG: --config-dir="/kube-dns-config"
I1110 16:35:35.686152       1 server.go:113] FLAG: --config-map=""
I1110 16:35:35.686155       1 server.go:113] FLAG: --config-map-namespace="kube-system"
I1110 16:35:35.686158       1 server.go:113] FLAG: --config-period="10s"
I1110 16:35:35.686161       1 server.go:113] FLAG: --dns-bind-address="0.0.0.0"
I1110 16:35:35.686164       1 server.go:113] FLAG: --dns-port="10053"
I1110 16:35:35.686192       1 server.go:113] FLAG: --domain="cluster.local."
I1110 16:35:35.686196       1 server.go:113] FLAG: --federations=""
I1110 16:35:35.686200       1 server.go:113] FLAG: --healthz-port="8081"
I1110 16:35:35.686202       1 server.go:113] FLAG: --initial-sync-timeout="1m0s"
I1110 16:35:35.686205       1 server.go:113] FLAG: --kube-master-url=""
I1110 16:35:35.686208       1 server.go:113] FLAG: --kubecfg-file=""
I1110 16:35:35.686211       1 server.go:113] FLAG: --log-backtrace-at=":0"
I1110 16:35:35.686236       1 server.go:113] FLAG: --log-dir=""
I1110 16:35:35.686250       1 server.go:113] FLAG: --log-flush-frequency="5s"
I1110 16:35:35.686257       1 server.go:113] FLAG: --logtostderr="true"
I1110 16:35:35.686260       1 server.go:113] FLAG: --nameservers=""
I1110 16:35:35.686262       1 server.go:113] FLAG: --stderrthreshold="2"
I1110 16:35:35.686275       1 server.go:113] FLAG: --v="2"
I1110 16:35:35.686281       1 server.go:113] FLAG: --version="false"
I1110 16:35:35.686286       1 server.go:113] FLAG: --vmodule=""
I1110 16:35:35.686459       1 server.go:176] Starting SkyDNS server (0.0.0.0:10053)
I1110 16:35:35.686713       1 server.go:198] Skydns metrics enabled (/metrics:10055)
I1110 16:35:35.686724       1 dns.go:147] Starting endpointsController
I1110 16:35:35.686728       1 dns.go:150] Starting serviceController
I1110 16:35:35.686924       1 logs.go:41] skydns: ready for queries on cluster.local. for tcp://0.0.0.0:10053 [rcache 0]
I1110 16:35:35.686937       1 logs.go:41] skydns: ready for queries on cluster.local. for udp://0.0.0.0:10053 [rcache 0]
I1110 16:35:36.187196       1 dns.go:171] Initialized services and endpoints from apiserver
I1110 16:35:36.187215       1 server.go:129] Setting up Healthz Handler (/readiness)
I1110 16:35:36.187221       1 server.go:134] Setting up cache handler (/cache)
I1110 16:35:36.187226       1 server.go:120] Status HTTP port 8081

The pod I am testing this against is setup to use ClusterFirst DNS type which is what the docs say I need for this to work.

Any advice or pointers as to where to look for the issue would be most welcome.

Solution

I also read those blogs/tutorials and they seem to fail to mention that you need to run kubectl apply -f your-dns-configmap.yaml on your config map.

I'd run the following: kubectl get cm --namespace="kube-system" to see if your config map is actually there or not. It should turn up with the name kube-dns.

I'm running into another issue and your post inspired me to check the kube-dns logs, at the top of my logs it says: 1 server.go:66] Using configuration read from ConfigMap: kube-system:kube-dns though your logs are saying Using configuration read from directory: /kube-dns-config.

So it appears your kube-dns is definitely not finding the config map and is defaulting to some config in the kube-dns containers filesystem. I'd definitely try run that apply and get cm command and then try again to see what happens.