Search code examples
dockercrashkubernetesunauthorizedkube-dns

Kubernetes on docker creates containers with empty serviceaccount and no tokens leading to container crash and restarts

Still experiencing a similar issues 1.3.0 and upto 1.4.0-alpha.0

In my case (docker based set up), either trusty or kubedns would get unauthorized from api server.

and strangely I see that the secrets are not there inside the instances, under the path /var/run/secrets/kubernetes.io/serviceaccount 

[root@ ... ]#  kubectl exec -it kube-dns-v13-htfjo ls /bin/sh
/ #
/ # ls /var/run/secrets/kubernetes.io/serviceaccount
/ #

While it seems they are in the node and in the proxy instance 

tmpfs on /var/lib/kubelet/pods/3de53b0c-45bb-11e6-9f03-08002776167a/volumes/kubernetes.io~secret/default-token-8axd8 type
tmpfs on /var/lib/kubelet/pods/3de5591e-45bb-11e6-9f03-08002776167a/volumes/kubernetes.io~secret/default-token-8axd8 type
tmpfs on /var/lib/kubelet/pods/f29f35c7-45cc-11e6-9f03-08002776167a/volumes/kubernetes.io~secret/default-token-ql88q type
  • Deleting the secret and deleting the pods then recreating them has no effect
  • Restarting cluster after unmounting & deleting the folders has no effect either

Naturally this results in kubedns unable to start. Log below 

I0709 09:04:11.578816       1 dns.go:394] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0709 09:04:11.578873       1 dns.go:427] records:[], retval:[], path:[local cluster svc default kubernetes]
I0709 09:04:11.579657       1 dns.go:394] Received DNS Request:kubernetes.default.svc.cluster.local., exact:false
I0709 09:04:11.579677       1 dns.go:427] records:[], retval:[], path:[local cluster svc default kubernetes]
E0709 09:04:11.786646       1 reflector.go:216] pkg/dns/dns.go:128: Failed to list *api.Service: serializer for text/html; charset=utf-8 doesn't exist
E0709 09:04:11.786995       1 reflector.go:216] pkg/dns/dns.go:127: Failed to list *api.Endpoints: serializer for text/html; charset=utf-8 doesn't exist
I0709 09:04:12.488674       1 dns.go:145] Ignoring error while waiting for service default/kubernetes: serializer for text/html; charset=utf-8 doesn't exist. Sleeping 1s before retrying.
E0709 09:04:12.879701       1 reflector.go:216] pkg/dns/dns.go:128: Failed to list *api.Service: serializer for text/html; charset=utf-8 doesn't exist
E0709 09:04:12.880000       1 reflector.go:216] pkg/dns/dns.go:127: Failed to list *api.Endpoints: serializer for text/html; charset=utf-8 doesn't exist
I0709 09:04:13.582561       1 dns.go:145] Ignoring error while waiting for service default/kubernetes: serializer for text/html; charset=utf-8 doesn't exist. Sleeping 1s before retrying.
Solution

  • This one seems to be a bug still open

    https://github.com/kubernetes/kubernetes/issues/26943

    The workaround that works is to add rslave option in the kubelet mount like --volume=/var/lib/kubelet:/var/lib/kubelet:rw,rslave as shown below.

    This solution also is platform dependent. Read the notes in the bug report. 

    ## Start kubernetes master
sudo docker run \
--volume=/:/rootfs:ro \
--volume=/sys:/sys:ro \
--volume=/var/lib/docker/:/var/lib/docker:rw \
**--volume=/var/lib/kubelet:/var/lib/kubelet:rw,rslave** \
--volume=/var/run:/var/run:rw \
--net=host \
--privileged=true \
--pid=host \
-d \
gcr.io/google_containers/hyperkube-amd64:${K8S_VERSION} \
/hyperkube kubelet \
    --allow-privileged=true \
    --api-servers=http://localhost:8080 \
    --v=2 \
    --address=0.0.0.0 \
    --enable-server \
    --hostname-override=127.0.0.1 \
    --config=/etc/kubernetes/manifests-multi \
    --containerized \
    --cluster-dns=10.0.0.10 \
    --cluster-domain=cluster.local